Sep 09 15:20:34.261051 ip-10-0-129-170 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 15:20:34.682894 ip-10-0-129-170 kubenswrapper[2553]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 15:20:34.682894 ip-10-0-129-170 kubenswrapper[2553]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 15:20:34.682894 ip-10-0-129-170 kubenswrapper[2553]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 15:20:34.682894 ip-10-0-129-170 kubenswrapper[2553]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 15:20:34.682894 ip-10-0-129-170 kubenswrapper[2553]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 15:20:34.684408 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.684318    2553 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 15:20:34.688145 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688129    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 15:20:34.688145 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688145    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 15:20:34.688209 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688149    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 15:20:34.688209 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688154    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 15:20:34.688209 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688157    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 15:20:34.688209 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688161    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 15:20:34.688209 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688164    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 15:20:34.688209 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688166    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 15:20:34.688209 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688169    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 15:20:34.688209 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688172    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 15:20:34.688209 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688175    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 15:20:34.688209 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688178    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 15:20:34.688209 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688181    2553 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 15:20:34.688209 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688184    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 15:20:34.688209 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688186    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 15:20:34.688209 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688191    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 15:20:34.688209 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688195    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 15:20:34.688209 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688198    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 15:20:34.688209 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688201    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 15:20:34.688209 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688203    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 15:20:34.688209 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688208    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 15:20:34.688666 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688211    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 15:20:34.688666 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688215    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 15:20:34.688666 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688217    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 15:20:34.688666 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688221    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 15:20:34.688666 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688224    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 15:20:34.688666 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688227    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 15:20:34.688666 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688229    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 15:20:34.688666 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688232    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 15:20:34.688666 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688235    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 15:20:34.688666 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688237    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 15:20:34.688666 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688240    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 15:20:34.688666 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688244    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 15:20:34.688666 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688246    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 15:20:34.688666 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688249    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 15:20:34.688666 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688252    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 15:20:34.688666 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688254    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 15:20:34.688666 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688257    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 15:20:34.688666 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688260    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 15:20:34.688666 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688263    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 15:20:34.688666 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688266    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 15:20:34.689205 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688268    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 15:20:34.689205 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688271    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 15:20:34.689205 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688275    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 15:20:34.689205 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688278    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 15:20:34.689205 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688280    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 15:20:34.689205 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688283    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 15:20:34.689205 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688285    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 15:20:34.689205 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688288    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 15:20:34.689205 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688290    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 15:20:34.689205 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688293    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 15:20:34.689205 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688297    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 15:20:34.689205 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688300    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 15:20:34.689205 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688303    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 15:20:34.689205 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688306    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 15:20:34.689205 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688310    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 15:20:34.689205 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688313    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 15:20:34.689205 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688316    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 15:20:34.689205 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688318    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 15:20:34.689205 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688321    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 15:20:34.689205 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688324    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 15:20:34.689715 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688327    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 15:20:34.689715 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688329    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 15:20:34.689715 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688332    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 15:20:34.689715 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688334    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 15:20:34.689715 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688337    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 15:20:34.689715 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688340    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 15:20:34.689715 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688343    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 15:20:34.689715 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688346    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 15:20:34.689715 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688348    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 15:20:34.689715 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688351    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 15:20:34.689715 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688354    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 15:20:34.689715 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688356    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 15:20:34.689715 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688359    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 15:20:34.689715 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688361    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 15:20:34.689715 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688364    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 15:20:34.689715 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688368    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 15:20:34.689715 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688373    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 15:20:34.689715 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688376    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 15:20:34.689715 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688379    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 15:20:34.690543 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688381    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 15:20:34.690543 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688384    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 15:20:34.690543 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688387    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 15:20:34.690543 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688389    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 15:20:34.690543 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688392    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 15:20:34.690543 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688395    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 15:20:34.690543 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688399    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 15:20:34.690543 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688402    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 15:20:34.690543 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688405    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 15:20:34.690543 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688408    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 15:20:34.690543 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688410    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 15:20:34.690543 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688413    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 15:20:34.690543 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688415    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 15:20:34.690543 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688418    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 15:20:34.690543 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688421    2553 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 15:20:34.690543 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688423    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 15:20:34.690543 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688425    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 15:20:34.690543 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688428    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 15:20:34.690543 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.688431    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 15:20:34.690543 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689131    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 15:20:34.691268 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689137    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 15:20:34.691268 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689140    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 15:20:34.691268 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689143    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 15:20:34.691268 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689146    2553 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 15:20:34.691268 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689148    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 15:20:34.691268 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689153    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 15:20:34.691268 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689156    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 15:20:34.691268 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689160    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 15:20:34.691268 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689163    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 15:20:34.691268 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689167    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 15:20:34.691268 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689171    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 15:20:34.691268 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689174    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 15:20:34.691268 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689176    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 15:20:34.691268 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689180    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 15:20:34.691268 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689182    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 15:20:34.691268 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689185    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 15:20:34.691268 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689188    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 15:20:34.691268 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689192    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 15:20:34.691268 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689196    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 15:20:34.691757 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689200    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 15:20:34.691757 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689203    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 15:20:34.691757 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689206    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 15:20:34.691757 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689209    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 15:20:34.691757 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689212    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 15:20:34.691757 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689215    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 15:20:34.691757 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689218    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 15:20:34.691757 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689221    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 15:20:34.691757 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689223    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 15:20:34.691757 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689226    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 15:20:34.691757 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689229    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 15:20:34.691757 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689231    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 15:20:34.691757 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689234    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 15:20:34.691757 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689237    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 15:20:34.691757 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689239    2553 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 15:20:34.691757 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689241    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 15:20:34.691757 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689244    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 15:20:34.691757 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689246    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 15:20:34.691757 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689249    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 15:20:34.691757 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689251    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 15:20:34.692257 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689254    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 15:20:34.692257 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689257    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 15:20:34.692257 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689260    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 15:20:34.692257 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689262    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 15:20:34.692257 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689264    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 15:20:34.692257 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689267    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 15:20:34.692257 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689270    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 15:20:34.692257 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689272    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 15:20:34.692257 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689274    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 15:20:34.692257 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689277    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 15:20:34.692257 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689280    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 15:20:34.692257 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689283    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 15:20:34.692257 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689285    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 15:20:34.692257 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689288    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 15:20:34.692257 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689291    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 15:20:34.692257 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689293    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 15:20:34.692257 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689296    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 15:20:34.692257 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689299    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 15:20:34.692257 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689301    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 15:20:34.692717 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689304    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 15:20:34.692717 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689306    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 15:20:34.692717 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689309    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 15:20:34.692717 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689311    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 15:20:34.692717 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689313    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 15:20:34.692717 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689316    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 15:20:34.692717 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689318    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 15:20:34.692717 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689321    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 15:20:34.692717 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689323    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 15:20:34.692717 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689326    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 15:20:34.692717 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689328    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 15:20:34.692717 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689331    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 15:20:34.692717 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689334    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 15:20:34.692717 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689336    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 15:20:34.692717 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689338    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 15:20:34.692717 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689341    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 15:20:34.692717 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689344    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 15:20:34.692717 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689346    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 15:20:34.692717 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689349    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 15:20:34.692717 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689351    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 15:20:34.693223 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689355    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 15:20:34.693223 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689357    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 15:20:34.693223 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689360    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 15:20:34.693223 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689363    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 15:20:34.693223 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689365    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 15:20:34.693223 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689368    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 15:20:34.693223 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689370    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 15:20:34.693223 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689373    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 15:20:34.693223 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689376    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 15:20:34.693223 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689379    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 15:20:34.693223 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689381    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 15:20:34.693223 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689384    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 15:20:34.693223 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689386    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 15:20:34.693223 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689388    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 15:20:34.693223 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689391    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 15:20:34.693223 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689393    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 15:20:34.693223 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689396    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 15:20:34.693223 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689398    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 15:20:34.693223 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689401    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 15:20:34.693223 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.689403    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 15:20:34.693716 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.690804    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 15:20:34.693716 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.690819    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 15:20:34.693716 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.690825    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 15:20:34.693716 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.690830    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 15:20:34.693716 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.690835    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 15:20:34.693716 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.690840    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 15:20:34.693716 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.690844    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 15:20:34.693716 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.690849    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 15:20:34.693716 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.690853    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 15:20:34.693716 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.690866    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 15:20:34.693716 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.690873    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 15:20:34.693716 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.690878    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 15:20:34.693716 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.690882    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 15:20:34.693716 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.690887    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 15:20:34.693716 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.690891    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 15:20:34.693716 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.690896    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 15:20:34.693716 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.690901    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 15:20:34.693716 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.690905    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 15:20:34.693716 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.690910    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 15:20:34.693716 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.690914    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 15:20:34.694294 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.690918    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 15:20:34.694294 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.690928    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 15:20:34.694294 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.690932    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 15:20:34.694294 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.690937    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 15:20:34.694294 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.690941    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 15:20:34.694294 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.690945    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 15:20:34.694294 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.690950    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 15:20:34.694294 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.690954    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 15:20:34.694294 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.690958    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 15:20:34.694294 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.690963    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 15:20:34.694294 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.690967    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 15:20:34.694294 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.690971    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 15:20:34.694294 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691021    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 15:20:34.694294 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691040    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 15:20:34.694294 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691045    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 15:20:34.694294 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691063    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 15:20:34.694294 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691090    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 15:20:34.694294 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691100    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 15:20:34.694294 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691221    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 15:20:34.694294 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691229    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 15:20:34.694834 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691234    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 15:20:34.694834 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691242    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 15:20:34.694834 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691247    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 15:20:34.694834 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691250    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 15:20:34.694834 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691253    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 15:20:34.694834 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691256    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 15:20:34.694834 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691259    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 15:20:34.694834 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691262    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 15:20:34.694834 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691265    2553 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 15:20:34.694834 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691267    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 15:20:34.694834 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691270    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 15:20:34.694834 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691273    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 15:20:34.694834 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691275    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 15:20:34.694834 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691278    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 15:20:34.694834 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691282    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 15:20:34.694834 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691284    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 15:20:34.694834 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691288    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 15:20:34.694834 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691290    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 15:20:34.694834 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691293    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 15:20:34.695312 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691296    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 15:20:34.695312 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691299    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 15:20:34.695312 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691302    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 15:20:34.695312 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691304    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 15:20:34.695312 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691306    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 15:20:34.695312 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691310    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 15:20:34.695312 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691312    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 15:20:34.695312 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691315    2553 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 15:20:34.695312 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691318    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 15:20:34.695312 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691325    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 15:20:34.695312 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691335    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 15:20:34.695312 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691338    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 15:20:34.695312 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691342    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 15:20:34.695312 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691345    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 15:20:34.695312 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691348    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 15:20:34.695312 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691351    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 15:20:34.695312 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691353    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 15:20:34.695312 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691357    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 15:20:34.695312 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691360    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 15:20:34.695787 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691362    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 15:20:34.695787 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691365    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 15:20:34.695787 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691368    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 15:20:34.695787 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691370    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 15:20:34.695787 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691373    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 15:20:34.695787 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691376    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 15:20:34.695787 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691379    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 15:20:34.695787 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691381    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 15:20:34.695787 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691384    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 15:20:34.695787 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691386    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 15:20:34.695787 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691389    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 15:20:34.695787 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691392    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 15:20:34.695787 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691395    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 15:20:34.695787 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691397    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 15:20:34.695787 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691400    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 15:20:34.695787 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691403    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 15:20:34.695787 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691405    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 15:20:34.695787 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691408    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 15:20:34.695787 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691411    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 15:20:34.695787 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691413    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 15:20:34.696331 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.691416    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 15:20:34.696331 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691498    2553 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 15:20:34.696331 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691542    2553 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 15:20:34.696331 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691550    2553 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 15:20:34.696331 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691554    2553 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 15:20:34.696331 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691559    2553 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 15:20:34.696331 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691562    2553 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 15:20:34.696331 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691566    2553 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 15:20:34.696331 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691570    2553 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 15:20:34.696331 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691574    2553 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 15:20:34.696331 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691577    2553 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 15:20:34.696331 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691580    2553 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 15:20:34.696331 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691583    2553 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 15:20:34.696331 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691587    2553 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 15:20:34.696331 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691590    2553 flags.go:64] FLAG: --cgroup-root=""
Sep 09 15:20:34.696331 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691593    2553 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 15:20:34.696331 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691596    2553 flags.go:64] FLAG: --client-ca-file=""
Sep 09 15:20:34.696331 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691599    2553 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 15:20:34.696331 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691602    2553 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 15:20:34.696331 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691607    2553 flags.go:64] FLAG: --cluster-domain=""
Sep 09 15:20:34.696331 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691609    2553 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 15:20:34.696331 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691613    2553 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 15:20:34.696331 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691616    2553 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 15:20:34.696331 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691620    2553 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 15:20:34.696938 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691624    2553 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 15:20:34.696938 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691627    2553 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 15:20:34.696938 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691630    2553 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 15:20:34.696938 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691633    2553 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 15:20:34.696938 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691636    2553 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 15:20:34.696938 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691639    2553 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 15:20:34.696938 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691642    2553 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 15:20:34.696938 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691645    2553 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 15:20:34.696938 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691649    2553 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 15:20:34.696938 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691653    2553 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 15:20:34.696938 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691657    2553 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 15:20:34.696938 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691660    2553 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 15:20:34.696938 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691663    2553 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 15:20:34.696938 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691666    2553 flags.go:64] FLAG: --enable-server="true"
Sep 09 15:20:34.696938 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691669    2553 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 15:20:34.696938 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691674    2553 flags.go:64] FLAG: --event-burst="100"
Sep 09 15:20:34.696938 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691678    2553 flags.go:64] FLAG: --event-qps="50"
Sep 09 15:20:34.696938 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691681    2553 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 15:20:34.696938 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691684    2553 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 15:20:34.696938 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691687    2553 flags.go:64] FLAG: --eviction-hard=""
Sep 09 15:20:34.696938 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691690    2553 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 15:20:34.696938 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691693    2553 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 15:20:34.696938 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691696    2553 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 15:20:34.696938 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691699    2553 flags.go:64] FLAG: --eviction-soft=""
Sep 09 15:20:34.696938 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691702    2553 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 15:20:34.697542 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691705    2553 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 15:20:34.697542 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691708    2553 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 15:20:34.697542 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691711    2553 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 15:20:34.697542 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691714    2553 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 15:20:34.697542 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691717    2553 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 15:20:34.697542 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691720    2553 flags.go:64] FLAG: --feature-gates=""
Sep 09 15:20:34.697542 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691724    2553 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 15:20:34.697542 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691727    2553 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 15:20:34.697542 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691746    2553 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 15:20:34.697542 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691752    2553 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 15:20:34.697542 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691756    2553 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 15:20:34.697542 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691759    2553 flags.go:64] FLAG: --help="false"
Sep 09 15:20:34.697542 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691762    2553 flags.go:64] FLAG: --hostname-override="ip-10-0-129-170.ec2.internal"
Sep 09 15:20:34.697542 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691765    2553 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 15:20:34.697542 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691768    2553 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 15:20:34.697542 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691772    2553 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 15:20:34.697542 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691776    2553 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 15:20:34.697542 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691779    2553 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 15:20:34.697542 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691782    2553 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 15:20:34.697542 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691785    2553 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 15:20:34.697542 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691788    2553 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 15:20:34.697542 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691791    2553 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 15:20:34.697542 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691793    2553 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 15:20:34.698163 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691797    2553 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 15:20:34.698163 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691799    2553 flags.go:64] FLAG: --kube-reserved=""
Sep 09 15:20:34.698163 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691802    2553 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 15:20:34.698163 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691805    2553 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 15:20:34.698163 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691808    2553 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 15:20:34.698163 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691811    2553 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 15:20:34.698163 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691814    2553 flags.go:64] FLAG: --lock-file=""
Sep 09 15:20:34.698163 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691816    2553 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 15:20:34.698163 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691819    2553 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 15:20:34.698163 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691822    2553 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 15:20:34.698163 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691828    2553 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 15:20:34.698163 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691831    2553 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 15:20:34.698163 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691835    2553 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 15:20:34.698163 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691838    2553 flags.go:64] FLAG: --logging-format="text"
Sep 09 15:20:34.698163 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691841    2553 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 15:20:34.698163 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691845    2553 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 15:20:34.698163 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691848    2553 flags.go:64] FLAG: --manifest-url=""
Sep 09 15:20:34.698163 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691850    2553 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 15:20:34.698163 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691855    2553 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 15:20:34.698163 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691858    2553 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 15:20:34.698163 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691862    2553 flags.go:64] FLAG: --max-pods="110"
Sep 09 15:20:34.698163 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691865    2553 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 15:20:34.698163 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691868    2553 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 15:20:34.698163 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691871    2553 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 15:20:34.698163 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691882    2553 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 15:20:34.698795 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691886    2553 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 15:20:34.698795 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691889    2553 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 15:20:34.698795 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691892    2553 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 15:20:34.698795 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691899    2553 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 15:20:34.698795 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691902    2553 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 15:20:34.698795 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691906    2553 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 15:20:34.698795 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691909    2553 flags.go:64] FLAG: --pod-cidr=""
Sep 09 15:20:34.698795 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691912    2553 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 15:20:34.698795 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691917    2553 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 15:20:34.698795 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691919    2553 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 15:20:34.698795 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691923    2553 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 15:20:34.698795 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691925    2553 flags.go:64] FLAG: --port="10250"
Sep 09 15:20:34.698795 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691928    2553 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 15:20:34.698795 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691931    2553 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-043a479f7b6a5830c"
Sep 09 15:20:34.698795 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691935    2553 flags.go:64] FLAG: --qos-reserved=""
Sep 09 15:20:34.698795 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691938    2553 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 15:20:34.698795 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691941    2553 flags.go:64] FLAG: --register-node="true"
Sep 09 15:20:34.698795 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691944    2553 flags.go:64] FLAG: --register-with-taints=""
Sep 09 15:20:34.698795 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691948    2553 flags.go:64] FLAG: --registry-burst="10"
Sep 09 15:20:34.698795 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691951    2553 flags.go:64] FLAG: --registry-qps="5"
Sep 09 15:20:34.698795 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691954    2553 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 15:20:34.698795 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691957    2553 flags.go:64] FLAG: --reserved-memory=""
Sep 09 15:20:34.698795 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691961    2553 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 15:20:34.698795 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691964    2553 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 15:20:34.698795 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691970    2553 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 15:20:34.699417 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691973    2553 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 15:20:34.699417 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691976    2553 flags.go:64] FLAG: --runonce="false"
Sep 09 15:20:34.699417 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691979    2553 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 15:20:34.699417 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691982    2553 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 15:20:34.699417 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691985    2553 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 15:20:34.699417 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691988    2553 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 15:20:34.699417 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691991    2553 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 15:20:34.699417 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691994    2553 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 15:20:34.699417 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.691997    2553 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 15:20:34.699417 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.692000    2553 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 15:20:34.699417 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.692003    2553 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 15:20:34.699417 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.692006    2553 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 15:20:34.699417 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.692009    2553 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 15:20:34.699417 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.692012    2553 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 15:20:34.699417 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.692015    2553 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 15:20:34.699417 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.692018    2553 flags.go:64] FLAG: --system-cgroups=""
Sep 09 15:20:34.699417 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.692021    2553 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 15:20:34.699417 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.692027    2553 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 15:20:34.699417 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.692030    2553 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 15:20:34.699417 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.692033    2553 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 15:20:34.699417 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.692037    2553 flags.go:64] FLAG: --tls-min-version=""
Sep 09 15:20:34.699417 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.692040    2553 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 15:20:34.699417 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.692043    2553 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 15:20:34.699417 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.692046    2553 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 15:20:34.699417 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.692049    2553 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 15:20:34.700084 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.692052    2553 flags.go:64] FLAG: --v="2"
Sep 09 15:20:34.700084 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.692056    2553 flags.go:64] FLAG: --version="false"
Sep 09 15:20:34.700084 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.692060    2553 flags.go:64] FLAG: --vmodule=""
Sep 09 15:20:34.700084 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.692065    2553 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 15:20:34.700084 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.692068    2553 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 15:20:34.700084 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692184    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 15:20:34.700084 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692189    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 15:20:34.700084 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692193    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 15:20:34.700084 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692195    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 15:20:34.700084 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692198    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 15:20:34.700084 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692201    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 15:20:34.700084 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692203    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 15:20:34.700084 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692206    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 15:20:34.700084 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692209    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 15:20:34.700084 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692212    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 15:20:34.700084 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692216    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 15:20:34.700084 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692219    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 15:20:34.700084 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692223    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 15:20:34.700084 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692226    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 15:20:34.700084 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692228    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 15:20:34.700573 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692231    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 15:20:34.700573 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692233    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 15:20:34.700573 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692236    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 15:20:34.700573 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692239    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 15:20:34.700573 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692241    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 15:20:34.700573 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692244    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 15:20:34.700573 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692246    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 15:20:34.700573 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692249    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 15:20:34.700573 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692252    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 15:20:34.700573 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692254    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 15:20:34.700573 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692257    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 15:20:34.700573 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692261    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 15:20:34.700573 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692266    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 15:20:34.700573 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692270    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 15:20:34.700573 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692273    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 15:20:34.700573 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692275    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 15:20:34.700573 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692278    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 15:20:34.700573 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692281    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 15:20:34.700573 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692285    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 15:20:34.701059 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692287    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 15:20:34.701059 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692290    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 15:20:34.701059 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692292    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 15:20:34.701059 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692295    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 15:20:34.701059 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692298    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 15:20:34.701059 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692300    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 15:20:34.701059 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692303    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 15:20:34.701059 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692305    2553 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 15:20:34.701059 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692308    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 15:20:34.701059 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692310    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 15:20:34.701059 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692313    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 15:20:34.701059 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692315    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 15:20:34.701059 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692319    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 15:20:34.701059 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692323    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 15:20:34.701059 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692326    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 15:20:34.701059 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692329    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 15:20:34.701059 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692331    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 15:20:34.701059 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692334    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 15:20:34.701059 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692337    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 15:20:34.701059 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692339    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 15:20:34.701542 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692342    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 15:20:34.701542 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692344    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 15:20:34.701542 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692347    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 15:20:34.701542 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692350    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 15:20:34.701542 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692352    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 15:20:34.701542 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692356    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 15:20:34.701542 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692359    2553 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 15:20:34.701542 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692361    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 15:20:34.701542 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692364    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 15:20:34.701542 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692366    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 15:20:34.701542 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692370    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 15:20:34.701542 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692374    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 15:20:34.701542 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692376    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 15:20:34.701542 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692379    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 15:20:34.701542 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692381    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 15:20:34.701542 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692384    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 15:20:34.701542 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692387    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 15:20:34.701542 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692389    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 15:20:34.701542 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692392    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 15:20:34.701542 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692394    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 15:20:34.702044 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692397    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 15:20:34.702044 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692399    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 15:20:34.702044 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692402    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 15:20:34.702044 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692405    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 15:20:34.702044 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692407    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 15:20:34.702044 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692410    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 15:20:34.702044 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692413    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 15:20:34.702044 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692415    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 15:20:34.702044 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692418    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 15:20:34.702044 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692421    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 15:20:34.702044 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692423    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 15:20:34.702044 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692426    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 15:20:34.702044 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692428    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 15:20:34.702044 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692431    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 15:20:34.702044 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692433    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 15:20:34.702044 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692436    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 15:20:34.702044 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692438    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 15:20:34.702044 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692443    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 15:20:34.702044 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692446    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 15:20:34.702044 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692448    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 15:20:34.702535 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692451    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 15:20:34.702535 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692453    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 15:20:34.702535 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692456    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 15:20:34.702535 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692460    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 15:20:34.702535 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.692463    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 15:20:34.702535 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.692468    2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 15:20:34.702535 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.699577    2553 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 15:20:34.702535 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.699592    2553 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 15:20:34.702535 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.700363    2553 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 15:20:34.702535 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.700371    2553 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 15:20:34.702535 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700427    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 15:20:34.702535 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700433    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 15:20:34.702535 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700438    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 15:20:34.702535 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700441    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 15:20:34.702901 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700444    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 15:20:34.702901 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700447    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 15:20:34.702901 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700450    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 15:20:34.702901 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700453    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 15:20:34.702901 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700457    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 15:20:34.702901 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700459    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 15:20:34.702901 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700462    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 15:20:34.702901 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700465    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 15:20:34.702901 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700467    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 15:20:34.702901 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700470    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 15:20:34.702901 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700474    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 15:20:34.702901 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700478    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 15:20:34.702901 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700481    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 15:20:34.702901 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700483    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 15:20:34.702901 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700486    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 15:20:34.702901 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700489    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 15:20:34.702901 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700492    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 15:20:34.702901 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700495    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 15:20:34.702901 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700498    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 15:20:34.703396 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700501    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 15:20:34.703396 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700503    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 15:20:34.703396 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700506    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 15:20:34.703396 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700510    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 15:20:34.703396 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700512    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 15:20:34.703396 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700515    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 15:20:34.703396 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700520    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 15:20:34.703396 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700523    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 15:20:34.703396 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700525    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 15:20:34.703396 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700528    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 15:20:34.703396 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700531    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 15:20:34.703396 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700533    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 15:20:34.703396 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700536    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 15:20:34.703396 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700538    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 15:20:34.703396 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700541    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 15:20:34.703396 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700543    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 15:20:34.703396 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700546    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 15:20:34.703396 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700549    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 15:20:34.703396 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700551    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 15:20:34.703396 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700554    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 15:20:34.703902 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700556    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 15:20:34.703902 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700560    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 15:20:34.703902 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700564    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 15:20:34.703902 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700567    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 15:20:34.703902 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700570    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 15:20:34.703902 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700573    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 15:20:34.703902 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700577    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 15:20:34.703902 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700580    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 15:20:34.703902 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700583    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 15:20:34.703902 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700586    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 15:20:34.703902 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700589    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 15:20:34.703902 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700592    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 15:20:34.703902 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700594    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 15:20:34.703902 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700597    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 15:20:34.703902 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700600    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 15:20:34.703902 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700602    2553 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 15:20:34.703902 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700605    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 15:20:34.703902 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700608    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 15:20:34.703902 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700611    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 15:20:34.704364 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700614    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 15:20:34.704364 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700616    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 15:20:34.704364 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700619    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 15:20:34.704364 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700621    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 15:20:34.704364 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700624    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 15:20:34.704364 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700626    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 15:20:34.704364 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700629    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 15:20:34.704364 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700631    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 15:20:34.704364 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700634    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 15:20:34.704364 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700637    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 15:20:34.704364 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700639    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 15:20:34.704364 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700641    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 15:20:34.704364 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700644    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 15:20:34.704364 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700647    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 15:20:34.704364 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700649    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 15:20:34.704364 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700652    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 15:20:34.704364 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700655    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 15:20:34.704364 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700657    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 15:20:34.704364 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700660    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 15:20:34.704364 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700663    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 15:20:34.704908 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700666    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 15:20:34.704908 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700668    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 15:20:34.704908 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700671    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 15:20:34.704908 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700673    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 15:20:34.704908 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700676    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 15:20:34.704908 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700679    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 15:20:34.704908 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700681    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 15:20:34.704908 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700684    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 15:20:34.704908 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700686    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 15:20:34.704908 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700689    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 15:20:34.704908 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700692    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 15:20:34.704908 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700694    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 15:20:34.704908 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700697    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 15:20:34.704908 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700700    2553 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 15:20:34.704908 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700703    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 15:20:34.704908 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700706    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 15:20:34.704908 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700708    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 15:20:34.705317 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.700713    2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 15:20:34.705317 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700843    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 15:20:34.705317 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700849    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 15:20:34.705317 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700852    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 15:20:34.705317 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700855    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 15:20:34.705317 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700858    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 15:20:34.705317 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700861    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 15:20:34.705317 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700864    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 15:20:34.705317 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700866    2553 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 15:20:34.705317 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700869    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 15:20:34.705317 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700872    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 15:20:34.705317 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700875    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 15:20:34.705317 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700877    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 15:20:34.705317 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700880    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 15:20:34.705317 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700883    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 15:20:34.705686 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700886    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 15:20:34.705686 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700888    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 15:20:34.705686 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700891    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 15:20:34.705686 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700893    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 15:20:34.705686 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700896    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 15:20:34.705686 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700899    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 15:20:34.705686 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700901    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 15:20:34.705686 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700904    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 15:20:34.705686 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700907    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 15:20:34.705686 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700910    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 15:20:34.705686 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700913    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 15:20:34.705686 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700915    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 15:20:34.705686 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700918    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 15:20:34.705686 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700921    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 15:20:34.705686 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700924    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 15:20:34.705686 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700927    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 15:20:34.705686 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700929    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 15:20:34.705686 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700932    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 15:20:34.705686 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700935    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 15:20:34.706153 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700939    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 15:20:34.706153 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700942    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 15:20:34.706153 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700945    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 15:20:34.706153 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700948    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 15:20:34.706153 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700951    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 15:20:34.706153 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700954    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 15:20:34.706153 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700957    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 15:20:34.706153 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700959    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 15:20:34.706153 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700962    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 15:20:34.706153 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700965    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 15:20:34.706153 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700968    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 15:20:34.706153 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700970    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 15:20:34.706153 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700974    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 15:20:34.706153 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700976    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 15:20:34.706153 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700979    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 15:20:34.706153 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700982    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 15:20:34.706153 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700987    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 15:20:34.706153 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700990    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 15:20:34.706153 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700992    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 15:20:34.706153 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700995    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 15:20:34.706630 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.700998    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 15:20:34.706630 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701001    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 15:20:34.706630 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701004    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 15:20:34.706630 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701006    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 15:20:34.706630 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701009    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 15:20:34.706630 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701011    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 15:20:34.706630 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701014    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 15:20:34.706630 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701017    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 15:20:34.706630 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701020    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 15:20:34.706630 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701022    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 15:20:34.706630 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701025    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 15:20:34.706630 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701028    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 15:20:34.706630 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701032    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 15:20:34.706630 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701035    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 15:20:34.706630 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701038    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 15:20:34.706630 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701041    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 15:20:34.706630 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701044    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 15:20:34.706630 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701047    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 15:20:34.706630 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701050    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 15:20:34.707165 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701052    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 15:20:34.707165 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701055    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 15:20:34.707165 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701058    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 15:20:34.707165 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701060    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 15:20:34.707165 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701063    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 15:20:34.707165 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701066    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 15:20:34.707165 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701069    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 15:20:34.707165 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701072    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 15:20:34.707165 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701074    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 15:20:34.707165 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701078    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 15:20:34.707165 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701081    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 15:20:34.707165 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701083    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 15:20:34.707165 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701086    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 15:20:34.707165 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701088    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 15:20:34.707165 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701091    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 15:20:34.707165 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701095    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 15:20:34.707165 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701098    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 15:20:34.707165 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701100    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 15:20:34.707165 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701103    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 15:20:34.707640 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701106    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 15:20:34.707640 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701108    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 15:20:34.707640 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701111    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 15:20:34.707640 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701114    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 15:20:34.707640 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701116    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 15:20:34.707640 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701119    2553 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 15:20:34.707640 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701122    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 15:20:34.707640 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:34.701124    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 15:20:34.707640 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.701129    2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 15:20:34.707640 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.701243    2553 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 15:20:34.707640 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.703202    2553 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 15:20:34.707640 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.704288    2553 server.go:1019] "Starting client certificate rotation"
Sep 09 15:20:34.707640 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.704390    2553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 15:20:34.707640 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.704453    2553 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 15:20:34.708022 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.706852    2553 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 15:20:34.708022 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.707991    2553 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 15:20:34.721702 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.721683    2553 log.go:25] "Validated CRI v1 runtime API"
Sep 09 15:20:34.725109 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.725092    2553 log.go:25] "Validated CRI v1 image API"
Sep 09 15:20:34.725953 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.725935    2553 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 15:20:34.728887 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.728863    2553 fs.go:135] Filesystem UUIDs: map[4df728b2-9c74-4bef-b0fc-ff994d3d8595:/dev/nvme0n1p4 4f701efb-d7a7-4ba6-a94c-9e39d069c164:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 09 15:20:34.728946 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.728885    2553 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 15:20:34.734483 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.734369    2553 manager.go:216] Machine: {Timestamp:2026-09-09 15:20:34.733134284 +0000 UTC m=+0.332028919 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3059520 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28df927456b9b63f6c5286dca93420 SystemUUID:ec28df92-7456-b9b6-3f6c-5286dca93420 BootID:29f848f4-a5f1-403e-826b-90271ca6959b Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:04:88:b4:65:e9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:04:88:b4:65:e9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3e:33:1f:c2:ed:68 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 15:20:34.734483 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.734479    2553 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 15:20:34.734613 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.734601    2553 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 15:20:34.736404 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.736388    2553 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 15:20:34.737713 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.737688    2553 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 15:20:34.737865 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.737718    2553 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-170.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 15:20:34.739110 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.739098    2553 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 15:20:34.739148 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.739115    2553 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 15:20:34.739148 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.739124    2553 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 15:20:34.739694 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.739684    2553 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 15:20:34.740997 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.740987    2553 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 15:20:34.745585 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.745572    2553 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 15:20:34.745710 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.745700    2553 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 15:20:34.745751 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.745743    2553 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 15:20:34.745780 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.745753    2553 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 15:20:34.745780 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.745768    2553 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 15:20:34.745780 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.745777    2553 kubelet.go:398] "Adding apiserver pod source"
Sep 09 15:20:34.745851 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.745787    2553 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 15:20:34.746250 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.746235    2553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 15:20:34.746334 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.746261    2553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 15:20:34.748574 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.748556    2553 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 15:20:34.750786 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.750772    2553 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 15:20:34.750837 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.750802    2553 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 15:20:34.750981 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.750971    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 15:20:34.751023 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.750986    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 15:20:34.751023 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.750992    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 15:20:34.751023 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.750998    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 15:20:34.751023 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.751006    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 15:20:34.751023 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.751015    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 15:20:34.751721 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.751709    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 15:20:34.751721 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.751719    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 15:20:34.751825 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.751726    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 15:20:34.751825 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.751747    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 15:20:34.751825 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.751756    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 15:20:34.751825 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.751765    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 15:20:34.752564 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.752554    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 15:20:34.752610 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.752566    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 15:20:34.755139 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.755120    2553 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h848x"
Sep 09 15:20:34.756660 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.756639    2553 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-170.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 15:20:34.756660 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:34.756664    2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-170.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 15:20:34.756808 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:34.756672    2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 15:20:34.756957 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.756944    2553 server.go:1268] "Started kubelet"
Sep 09 15:20:34.757171 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.757135    2553 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 15:20:34.757171 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.757133    2553 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 15:20:34.757316 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.757199    2553 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 15:20:34.757700 ip-10-0-129-170 systemd[1]: Started Kubernetes Kubelet.
Sep 09 15:20:34.758305 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.758288    2553 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 15:20:34.760452 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.760433    2553 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 15:20:34.762943 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.762919    2553 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h848x"
Sep 09 15:20:34.763357 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:34.763339    2553 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 15:20:34.763447 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.763428    2553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 15:20:34.764041 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.764024    2553 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 15:20:34.764340 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:34.764261    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-170.ec2.internal\" not found"
Sep 09 15:20:34.764677 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.764659    2553 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 15:20:34.764776 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.764680    2553 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 15:20:34.765532 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.765468    2553 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 15:20:34.765618 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:34.763949    2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-170.ec2.internal.18d3b038be7fc6d7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-170.ec2.internal,UID:ip-10-0-129-170.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-170.ec2.internal,},FirstTimestamp:2026-09-09 15:20:34.756921047 +0000 UTC m=+0.355815682,LastTimestamp:2026-09-09 15:20:34.756921047 +0000 UTC m=+0.355815682,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-170.ec2.internal,}"
Sep 09 15:20:34.765885 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.765872    2553 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 15:20:34.765961 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.765953    2553 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 15:20:34.766485 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.766438    2553 factory.go:153] Registering CRI-O factory
Sep 09 15:20:34.766578 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.766492    2553 factory.go:223] Registration of the crio container factory successfully
Sep 09 15:20:34.766578 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.766542    2553 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 15:20:34.766578 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.766553    2553 factory.go:55] Registering systemd factory
Sep 09 15:20:34.766578 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.766562    2553 factory.go:223] Registration of the systemd container factory successfully
Sep 09 15:20:34.766578 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.766579    2553 factory.go:103] Registering Raw factory
Sep 09 15:20:34.766787 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.766595    2553 manager.go:1196] Started watching for new ooms in manager
Sep 09 15:20:34.767556 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.767538    2553 manager.go:349] Starting recovery of all containers
Sep 09 15:20:34.772941 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.772921    2553 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 15:20:34.776591 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:34.776570    2553 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-170.ec2.internal\" not found" node="ip-10-0-129-170.ec2.internal"
Sep 09 15:20:34.777134 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.777116    2553 manager.go:354] Recovery completed
Sep 09 15:20:34.781459 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.781445    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 15:20:34.783235 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.783221    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-170.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 15:20:34.783304 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.783250    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-170.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 15:20:34.783304 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.783260    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-170.ec2.internal" event="NodeHasSufficientPID"
Sep 09 15:20:34.783988 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.783973    2553 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 15:20:34.783988 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.783986    2553 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 15:20:34.784131 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.783992    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 15:20:34.784131 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.784002    2553 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 15:20:34.784703 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.784687    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-170.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 15:20:34.784814 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.784710    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-170.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 15:20:34.784814 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.784721    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-170.ec2.internal" event="NodeHasSufficientPID"
Sep 09 15:20:34.785879 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.785868    2553 policy_none.go:49] "None policy: Start"
Sep 09 15:20:34.785915 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.785887    2553 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 15:20:34.785915 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.785897    2553 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 15:20:34.787679 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.787663    2553 policy_none.go:47] "Start"
Sep 09 15:20:34.821378 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.821362    2553 manager.go:337] "Starting Device Plugin manager"
Sep 09 15:20:34.845245 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:34.821402    2553 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 15:20:34.845245 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.821416    2553 server.go:87] "Starting device plugin registration server"
Sep 09 15:20:34.845245 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.821699    2553 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 15:20:34.845245 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.821712    2553 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 15:20:34.845245 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.821848    2553 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 15:20:34.845245 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.821925    2553 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 15:20:34.845245 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.821932    2553 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 15:20:34.845245 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:34.822457    2553 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 15:20:34.845245 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:34.822505    2553 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-170.ec2.internal\" not found"
Sep 09 15:20:34.922804 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.922766    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 15:20:34.923838 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.923819    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-170.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 15:20:34.923919 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.923854    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-170.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 15:20:34.923919 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.923865    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-170.ec2.internal" event="NodeHasSufficientPID"
Sep 09 15:20:34.923919 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.923901    2553 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-170.ec2.internal"
Sep 09 15:20:34.931958 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.931939    2553 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-170.ec2.internal"
Sep 09 15:20:34.932039 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:34.931965    2553 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-170.ec2.internal\": node \"ip-10-0-129-170.ec2.internal\" not found"
Sep 09 15:20:34.948569 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:34.948494    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-170.ec2.internal\" not found"
Sep 09 15:20:34.954304 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.954269    2553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 15:20:34.955551 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.955531    2553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 15:20:34.955655 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.955580    2553 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 15:20:34.955655 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.955600    2553 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 15:20:34.955655 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:34.955635    2553 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 15:20:34.957520 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:34.957493    2553 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 15:20:35.048782 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:35.048728    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-170.ec2.internal\" not found"
Sep 09 15:20:35.055909 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.055882    2553 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-170.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-170.ec2.internal"]
Sep 09 15:20:35.056008 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.055980    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 15:20:35.057636 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.057619    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-170.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 15:20:35.057715 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.057651    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-170.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 15:20:35.057715 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.057662    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-170.ec2.internal" event="NodeHasSufficientPID"
Sep 09 15:20:35.059159 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.059147    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 15:20:35.059292 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.059277    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-170.ec2.internal"
Sep 09 15:20:35.059330 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.059318    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 15:20:35.059933 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.059917    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-170.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 15:20:35.060011 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.059935    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-170.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 15:20:35.060011 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.059949    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-170.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 15:20:35.060011 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.059955    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-170.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 15:20:35.060011 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.059965    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-170.ec2.internal" event="NodeHasSufficientPID"
Sep 09 15:20:35.060192 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.059966    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-170.ec2.internal" event="NodeHasSufficientPID"
Sep 09 15:20:35.061317 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.061301    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-170.ec2.internal"
Sep 09 15:20:35.061390 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.061325    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 15:20:35.062005 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.061988    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-170.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 15:20:35.062087 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.062016    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-170.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 15:20:35.062087 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.062029    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-170.ec2.internal" event="NodeHasSufficientPID"
Sep 09 15:20:35.068327 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.068306    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17b2b5583df839506650e89cca8d5eb6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-170.ec2.internal\" (UID: \"17b2b5583df839506650e89cca8d5eb6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-170.ec2.internal"
Sep 09 15:20:35.068404 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.068341    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f934fa7e403724f763f7c170a46e1bd9-config\") pod \"kube-apiserver-proxy-ip-10-0-129-170.ec2.internal\" (UID: \"f934fa7e403724f763f7c170a46e1bd9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-170.ec2.internal"
Sep 09 15:20:35.068404 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.068365    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/17b2b5583df839506650e89cca8d5eb6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-170.ec2.internal\" (UID: \"17b2b5583df839506650e89cca8d5eb6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-170.ec2.internal"
Sep 09 15:20:35.086479 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:35.086458    2553 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-170.ec2.internal\" not found" node="ip-10-0-129-170.ec2.internal"
Sep 09 15:20:35.091086 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:35.091070    2553 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-170.ec2.internal\" not found" node="ip-10-0-129-170.ec2.internal"
Sep 09 15:20:35.149512 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:35.149481    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-170.ec2.internal\" not found"
Sep 09 15:20:35.168839 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.168816    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/17b2b5583df839506650e89cca8d5eb6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-170.ec2.internal\" (UID: \"17b2b5583df839506650e89cca8d5eb6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-170.ec2.internal"
Sep 09 15:20:35.168937 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.168846    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17b2b5583df839506650e89cca8d5eb6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-170.ec2.internal\" (UID: \"17b2b5583df839506650e89cca8d5eb6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-170.ec2.internal"
Sep 09 15:20:35.168937 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.168863    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f934fa7e403724f763f7c170a46e1bd9-config\") pod \"kube-apiserver-proxy-ip-10-0-129-170.ec2.internal\" (UID: \"f934fa7e403724f763f7c170a46e1bd9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-170.ec2.internal"
Sep 09 15:20:35.168937 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.168891    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f934fa7e403724f763f7c170a46e1bd9-config\") pod \"kube-apiserver-proxy-ip-10-0-129-170.ec2.internal\" (UID: \"f934fa7e403724f763f7c170a46e1bd9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-170.ec2.internal"
Sep 09 15:20:35.168937 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.168916    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/17b2b5583df839506650e89cca8d5eb6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-170.ec2.internal\" (UID: \"17b2b5583df839506650e89cca8d5eb6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-170.ec2.internal"
Sep 09 15:20:35.169068 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.168931    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17b2b5583df839506650e89cca8d5eb6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-170.ec2.internal\" (UID: \"17b2b5583df839506650e89cca8d5eb6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-170.ec2.internal"
Sep 09 15:20:35.250259 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:35.250181    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-170.ec2.internal\" not found"
Sep 09 15:20:35.354718 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:35.354686    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-170.ec2.internal\" not found"
Sep 09 15:20:35.389286 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.389245    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-170.ec2.internal"
Sep 09 15:20:35.393910 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.393475    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-170.ec2.internal"
Sep 09 15:20:35.455974 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:35.455931    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-170.ec2.internal\" not found"
Sep 09 15:20:35.556448 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:35.556369    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-170.ec2.internal\" not found"
Sep 09 15:20:35.657381 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:35.657345    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-170.ec2.internal\" not found"
Sep 09 15:20:35.704113 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.704075    2553 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 15:20:35.704596 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.704269    2553 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 15:20:35.704596 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.704276    2553 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 15:20:35.757489 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:35.757457    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-170.ec2.internal\" not found"
Sep 09 15:20:35.769061 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.769028    2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 15:15:35 +0000 UTC" deadline="2028-04-18 01:59:57.026643718 +0000 UTC"
Sep 09 15:20:35.769061 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.769057    2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14074h39m21.257589661s"
Sep 09 15:20:35.857885 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:35.857859    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-170.ec2.internal\" not found"
Sep 09 15:20:35.880578 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.880549    2553 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 15:20:35.965918 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.965887    2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-170.ec2.internal"
Sep 09 15:20:35.976860 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.976838    2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 15:20:35.978700 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.978682    2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-170.ec2.internal"
Sep 09 15:20:35.981058 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:35.981017    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf934fa7e403724f763f7c170a46e1bd9.slice/crio-f26ba2ba0848ad0da60f13348f3976e56a39ee57e9a2722d11a5f05b880d009d WatchSource:0}: Error finding container f26ba2ba0848ad0da60f13348f3976e56a39ee57e9a2722d11a5f05b880d009d: Status 404 returned error can't find the container with id f26ba2ba0848ad0da60f13348f3976e56a39ee57e9a2722d11a5f05b880d009d
Sep 09 15:20:35.984993 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.984973    2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 15:20:35.985052 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:35.985013    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 15:20:36.199519 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.199425    2553 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 15:20:36.749321 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.749292    2553 apiserver.go:52] "Watching apiserver"
Sep 09 15:20:36.758375 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.758348    2553 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 15:20:36.758757 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.758734    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-thqt6","openshift-ovn-kubernetes/ovnkube-node-45pwd","kube-system/global-pull-secret-syncer-kcl4l","kube-system/konnectivity-agent-xjfqb","kube-system/kube-apiserver-proxy-ip-10-0-129-170.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhn2q","openshift-cluster-node-tuning-operator/tuned-fwsrs","openshift-dns/node-resolver-wgm9j","openshift-multus/multus-additional-cni-plugins-cmk65","openshift-multus/multus-bt4f2","openshift-image-registry/node-ca-txxcj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-170.ec2.internal","openshift-multus/network-metrics-daemon-v5kbj","openshift-network-diagnostics/network-check-target-xwx7m"]
Sep 09 15:20:36.762779 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.762756    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.763800 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.763771    2553 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 15:20:36.766475 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.766454    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-txxcj"
Sep 09 15:20:36.766615 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.766555    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ltvqf\""
Sep 09 15:20:36.767033 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.766918    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 15:20:36.767033 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.766922    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 15:20:36.768951 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.768931    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcl4l"
Sep 09 15:20:36.769053 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:36.769001    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcl4l" podUID="26591625-285a-4c93-b5be-7d0f4f0a0aae"
Sep 09 15:20:36.769904 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.769884    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 15:20:36.770069 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.770046    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 15:20:36.770172 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.770055    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hzmfh\""
Sep 09 15:20:36.770268 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.770255    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 15:20:36.771437 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.771417    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xjfqb"
Sep 09 15:20:36.773845 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.773825    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwx7m"
Sep 09 15:20:36.773962 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:36.773923    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xwx7m" podUID="d830519d-a525-4029-ae78-d30bb04e19e5"
Sep 09 15:20:36.773962 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.773957    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 15:20:36.774079 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.773996    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 15:20:36.774079 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.774007    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8n4kk\""
Sep 09 15:20:36.775844 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.775804    2553 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 15:20:36.776268 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.776254    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhn2q"
Sep 09 15:20:36.777181 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.777158    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/587d946b-e46a-4e99-84bf-8ef93be089c9-etc-sysctl-conf\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.777265 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.777195    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/587d946b-e46a-4e99-84bf-8ef93be089c9-host\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.777265 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.777220    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/587d946b-e46a-4e99-84bf-8ef93be089c9-tmp\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.777265 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.777247    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzkvx\" (UniqueName: \"kubernetes.io/projected/587d946b-e46a-4e99-84bf-8ef93be089c9-kube-api-access-zzkvx\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.777427 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.777293    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgmjh\" (UniqueName: \"kubernetes.io/projected/1aaf71f6-9195-4b50-a9a4-3cb85b10956a-kube-api-access-tgmjh\") pod \"node-ca-txxcj\" (UID: \"1aaf71f6-9195-4b50-a9a4-3cb85b10956a\") " pod="openshift-image-registry/node-ca-txxcj"
Sep 09 15:20:36.777427 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.777326    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/587d946b-e46a-4e99-84bf-8ef93be089c9-etc-modprobe-d\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.777427 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.777352    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/587d946b-e46a-4e99-84bf-8ef93be089c9-etc-kubernetes\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.777427 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.777382    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/587d946b-e46a-4e99-84bf-8ef93be089c9-etc-sysctl-d\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.777427 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.777407    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/587d946b-e46a-4e99-84bf-8ef93be089c9-sys\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.777776 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.777432    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1aaf71f6-9195-4b50-a9a4-3cb85b10956a-host\") pod \"node-ca-txxcj\" (UID: \"1aaf71f6-9195-4b50-a9a4-3cb85b10956a\") " pod="openshift-image-registry/node-ca-txxcj"
Sep 09 15:20:36.777776 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.777456    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1aaf71f6-9195-4b50-a9a4-3cb85b10956a-serviceca\") pod \"node-ca-txxcj\" (UID: \"1aaf71f6-9195-4b50-a9a4-3cb85b10956a\") " pod="openshift-image-registry/node-ca-txxcj"
Sep 09 15:20:36.777776 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.777480    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/587d946b-e46a-4e99-84bf-8ef93be089c9-run\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.777776 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.777503    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/587d946b-e46a-4e99-84bf-8ef93be089c9-lib-modules\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.777776 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.777526    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/587d946b-e46a-4e99-84bf-8ef93be089c9-etc-tuned\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.777776 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.777550    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/587d946b-e46a-4e99-84bf-8ef93be089c9-etc-sysconfig\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.777776 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.777573    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/587d946b-e46a-4e99-84bf-8ef93be089c9-etc-systemd\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.777776 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.777600    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/587d946b-e46a-4e99-84bf-8ef93be089c9-var-lib-kubelet\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.778807 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.778720    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 15:20:36.778807 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.778794    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cmk65"
Sep 09 15:20:36.778963 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.778817    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 15:20:36.778963 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.778824    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sh9tj\""
Sep 09 15:20:36.779072 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.778970    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 15:20:36.781200 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.781180    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5kbj"
Sep 09 15:20:36.781276 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:36.781253    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5kbj" podUID="57473bc0-f6d0-45f1-8954-93a7d2f4321b"
Sep 09 15:20:36.781566 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.781544    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 15:20:36.781667 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.781646    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6lh59\""
Sep 09 15:20:36.781802 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.781788    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 15:20:36.781895 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.781816    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 15:20:36.782619 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.782599    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 15:20:36.782704 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.782682    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 15:20:36.783766 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.783728    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.786194 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.786163    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 15:20:36.786300 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.786283    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.786511 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.786493    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 15:20:36.786975 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.786954    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tvrrz\""
Sep 09 15:20:36.787521 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.787498    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 15:20:36.787619 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.787530    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 15:20:36.787619 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.787547    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 15:20:36.787619 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.787582    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 15:20:36.789437 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.789418    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-thqt6"
Sep 09 15:20:36.789606 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.789589    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-czbt4\""
Sep 09 15:20:36.789712 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.789697    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 15:20:36.791800 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.791782    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 15:20:36.792054 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.792021    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 15:20:36.792587 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.792239    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-t2bzg\""
Sep 09 15:20:36.792664 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.792596    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 15:20:36.792723 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.792673    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wgm9j"
Sep 09 15:20:36.795647 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.795624    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 15:20:36.795776 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.795650    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 15:20:36.795776 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.795656    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nvh5t\""
Sep 09 15:20:36.799148 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.799126    2553 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-686zd"
Sep 09 15:20:36.806673 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.806659    2553 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-686zd"
Sep 09 15:20:36.866400 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.866373    2553 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 15:20:36.877969 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.877936    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d957458f-a014-49cb-9403-efba9f50e7f9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xhn2q\" (UID: \"d957458f-a014-49cb-9403-efba9f50e7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhn2q"
Sep 09 15:20:36.877969 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.877966    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d957458f-a014-49cb-9403-efba9f50e7f9-device-dir\") pod \"aws-ebs-csi-driver-node-xhn2q\" (UID: \"d957458f-a014-49cb-9403-efba9f50e7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhn2q"
Sep 09 15:20:36.878161 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.877988    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d957458f-a014-49cb-9403-efba9f50e7f9-etc-selinux\") pod \"aws-ebs-csi-driver-node-xhn2q\" (UID: \"d957458f-a014-49cb-9403-efba9f50e7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhn2q"
Sep 09 15:20:36.878161 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878004    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-host-var-lib-cni-multus\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.878161 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878026    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-host-var-lib-kubelet\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.878161 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878074    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-host-kubelet\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.878161 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878124    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3cfc8b11-7c62-46fc-8a5e-d5d889501cfd-hosts-file\") pod \"node-resolver-wgm9j\" (UID: \"3cfc8b11-7c62-46fc-8a5e-d5d889501cfd\") " pod="openshift-dns/node-resolver-wgm9j"
Sep 09 15:20:36.878161 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878145    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-cni-binary-copy\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.878376 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878167    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdtd6\" (UniqueName: \"kubernetes.io/projected/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-kube-api-access-bdtd6\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.878376 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878185    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/436fdcf3-5cd3-44e0-b5b8-3b878c58f6c7-host-slash\") pod \"iptables-alerter-thqt6\" (UID: \"436fdcf3-5cd3-44e0-b5b8-3b878c58f6c7\") " pod="openshift-network-operator/iptables-alerter-thqt6"
Sep 09 15:20:36.878376 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878205    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/587d946b-e46a-4e99-84bf-8ef93be089c9-etc-systemd\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.878376 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878237    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/587d946b-e46a-4e99-84bf-8ef93be089c9-var-lib-kubelet\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.878376 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878283    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/587d946b-e46a-4e99-84bf-8ef93be089c9-var-lib-kubelet\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.878376 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878303    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/587d946b-e46a-4e99-84bf-8ef93be089c9-etc-systemd\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.878376 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878309    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rjkr\" (UniqueName: \"kubernetes.io/projected/4a532bf5-55b6-46a3-8c3f-4f3af4920a8d-kube-api-access-8rjkr\") pod \"multus-additional-cni-plugins-cmk65\" (UID: \"4a532bf5-55b6-46a3-8c3f-4f3af4920a8d\") " pod="openshift-multus/multus-additional-cni-plugins-cmk65"
Sep 09 15:20:36.878376 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878340    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-etc-openvswitch\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.878376 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878364    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-host-cni-bin\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.878744 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878380    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-multus-cni-dir\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.878744 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878397    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4a532bf5-55b6-46a3-8c3f-4f3af4920a8d-cnibin\") pod \"multus-additional-cni-plugins-cmk65\" (UID: \"4a532bf5-55b6-46a3-8c3f-4f3af4920a8d\") " pod="openshift-multus/multus-additional-cni-plugins-cmk65"
Sep 09 15:20:36.878744 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878413    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d957458f-a014-49cb-9403-efba9f50e7f9-socket-dir\") pod \"aws-ebs-csi-driver-node-xhn2q\" (UID: \"d957458f-a014-49cb-9403-efba9f50e7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhn2q"
Sep 09 15:20:36.878744 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878430    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/587d946b-e46a-4e99-84bf-8ef93be089c9-etc-modprobe-d\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.878744 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878449    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/587d946b-e46a-4e99-84bf-8ef93be089c9-sys\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.878744 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878476    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1aaf71f6-9195-4b50-a9a4-3cb85b10956a-serviceca\") pod \"node-ca-txxcj\" (UID: \"1aaf71f6-9195-4b50-a9a4-3cb85b10956a\") " pod="openshift-image-registry/node-ca-txxcj"
Sep 09 15:20:36.878744 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878506    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vgvr\" (UniqueName: \"kubernetes.io/projected/b60f4dd4-700c-4674-83ea-65b3d382f104-kube-api-access-4vgvr\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.878744 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878518    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/587d946b-e46a-4e99-84bf-8ef93be089c9-sys\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.878744 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878532    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26591625-285a-4c93-b5be-7d0f4f0a0aae-original-pull-secret\") pod \"global-pull-secret-syncer-kcl4l\" (UID: \"26591625-285a-4c93-b5be-7d0f4f0a0aae\") " pod="kube-system/global-pull-secret-syncer-kcl4l"
Sep 09 15:20:36.878744 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878544    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/587d946b-e46a-4e99-84bf-8ef93be089c9-etc-modprobe-d\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.878744 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878561    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll8rs\" (UniqueName: \"kubernetes.io/projected/d830519d-a525-4029-ae78-d30bb04e19e5-kube-api-access-ll8rs\") pod \"network-check-target-xwx7m\" (UID: \"d830519d-a525-4029-ae78-d30bb04e19e5\") " pod="openshift-network-diagnostics/network-check-target-xwx7m"
Sep 09 15:20:36.878744 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878578    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-host-run-netns\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.878744 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878603    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-multus-conf-dir\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.878744 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878628    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/587d946b-e46a-4e99-84bf-8ef93be089c9-run\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.878744 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878651    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/587d946b-e46a-4e99-84bf-8ef93be089c9-lib-modules\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.878744 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878743    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/587d946b-e46a-4e99-84bf-8ef93be089c9-lib-modules\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.879617 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878749    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-host-cni-netd\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.879617 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878769    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/587d946b-e46a-4e99-84bf-8ef93be089c9-run\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.879617 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878788    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b60f4dd4-700c-4674-83ea-65b3d382f104-ovnkube-config\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.879617 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878815    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cfc8b11-7c62-46fc-8a5e-d5d889501cfd-tmp-dir\") pod \"node-resolver-wgm9j\" (UID: \"3cfc8b11-7c62-46fc-8a5e-d5d889501cfd\") " pod="openshift-dns/node-resolver-wgm9j"
Sep 09 15:20:36.879617 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878833    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-hostroot\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.879617 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878849    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4a532bf5-55b6-46a3-8c3f-4f3af4920a8d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cmk65\" (UID: \"4a532bf5-55b6-46a3-8c3f-4f3af4920a8d\") " pod="openshift-multus/multus-additional-cni-plugins-cmk65"
Sep 09 15:20:36.879617 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878873    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-host-run-netns\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.879617 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878902    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-run-systemd\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.879617 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878932    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d957458f-a014-49cb-9403-efba9f50e7f9-sys-fs\") pod \"aws-ebs-csi-driver-node-xhn2q\" (UID: \"d957458f-a014-49cb-9403-efba9f50e7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhn2q"
Sep 09 15:20:36.879617 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878958    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/587d946b-e46a-4e99-84bf-8ef93be089c9-tmp\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.879617 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.878984    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-run-openvswitch\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.879617 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.879006    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4a532bf5-55b6-46a3-8c3f-4f3af4920a8d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cmk65\" (UID: \"4a532bf5-55b6-46a3-8c3f-4f3af4920a8d\") " pod="openshift-multus/multus-additional-cni-plugins-cmk65"
Sep 09 15:20:36.879617 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.879028    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57473bc0-f6d0-45f1-8954-93a7d2f4321b-metrics-certs\") pod \"network-metrics-daemon-v5kbj\" (UID: \"57473bc0-f6d0-45f1-8954-93a7d2f4321b\") " pod="openshift-multus/network-metrics-daemon-v5kbj"
Sep 09 15:20:36.879617 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.879035    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1aaf71f6-9195-4b50-a9a4-3cb85b10956a-serviceca\") pod \"node-ca-txxcj\" (UID: \"1aaf71f6-9195-4b50-a9a4-3cb85b10956a\") " pod="openshift-image-registry/node-ca-txxcj"
Sep 09 15:20:36.879617 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.879043    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-system-cni-dir\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.879617 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.879115    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-host-var-lib-cni-bin\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.879617 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.879146    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-etc-kubernetes\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.880484 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.879173    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/26591625-285a-4c93-b5be-7d0f4f0a0aae-kubelet-config\") pod \"global-pull-secret-syncer-kcl4l\" (UID: \"26591625-285a-4c93-b5be-7d0f4f0a0aae\") " pod="kube-system/global-pull-secret-syncer-kcl4l"
Sep 09 15:20:36.880484 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.879209    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-run-ovn\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.880484 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.879247    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-host-run-ovn-kubernetes\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.880484 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.879274    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b60f4dd4-700c-4674-83ea-65b3d382f104-ovnkube-script-lib\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.880484 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.879285    2553 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 15:20:36.880484 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.879318    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-multus-socket-dir-parent\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.880484 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.879353    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-host-run-k8s-cni-cncf-io\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.880484 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.879391    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvznv\" (UniqueName: \"kubernetes.io/projected/436fdcf3-5cd3-44e0-b5b8-3b878c58f6c7-kube-api-access-lvznv\") pod \"iptables-alerter-thqt6\" (UID: \"436fdcf3-5cd3-44e0-b5b8-3b878c58f6c7\") " pod="openshift-network-operator/iptables-alerter-thqt6"
Sep 09 15:20:36.880484 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.879424    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzkvx\" (UniqueName: \"kubernetes.io/projected/587d946b-e46a-4e99-84bf-8ef93be089c9-kube-api-access-zzkvx\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.880484 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.879463    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgmjh\" (UniqueName: \"kubernetes.io/projected/1aaf71f6-9195-4b50-a9a4-3cb85b10956a-kube-api-access-tgmjh\") pod \"node-ca-txxcj\" (UID: \"1aaf71f6-9195-4b50-a9a4-3cb85b10956a\") " pod="openshift-image-registry/node-ca-txxcj"
Sep 09 15:20:36.880484 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.879493    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3a444239-19ad-490a-a006-0a43ce90c9ef-agent-certs\") pod \"konnectivity-agent-xjfqb\" (UID: \"3a444239-19ad-490a-a006-0a43ce90c9ef\") " pod="kube-system/konnectivity-agent-xjfqb"
Sep 09 15:20:36.880484 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.879526    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-node-log\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.880484 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.879670    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.880484 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.879708    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzd95\" (UniqueName: \"kubernetes.io/projected/3cfc8b11-7c62-46fc-8a5e-d5d889501cfd-kube-api-access-bzd95\") pod \"node-resolver-wgm9j\" (UID: \"3cfc8b11-7c62-46fc-8a5e-d5d889501cfd\") " pod="openshift-dns/node-resolver-wgm9j"
Sep 09 15:20:36.880484 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.879779    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7sx8\" (UniqueName: \"kubernetes.io/projected/d957458f-a014-49cb-9403-efba9f50e7f9-kube-api-access-x7sx8\") pod \"aws-ebs-csi-driver-node-xhn2q\" (UID: \"d957458f-a014-49cb-9403-efba9f50e7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhn2q"
Sep 09 15:20:36.880484 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.879806    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-multus-daemon-config\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.880484 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.879837    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/587d946b-e46a-4e99-84bf-8ef93be089c9-etc-kubernetes\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.881169 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.879863    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d957458f-a014-49cb-9403-efba9f50e7f9-registration-dir\") pod \"aws-ebs-csi-driver-node-xhn2q\" (UID: \"d957458f-a014-49cb-9403-efba9f50e7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhn2q"
Sep 09 15:20:36.881169 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.879887    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnn5b\" (UniqueName: \"kubernetes.io/projected/57473bc0-f6d0-45f1-8954-93a7d2f4321b-kube-api-access-lnn5b\") pod \"network-metrics-daemon-v5kbj\" (UID: \"57473bc0-f6d0-45f1-8954-93a7d2f4321b\") " pod="openshift-multus/network-metrics-daemon-v5kbj"
Sep 09 15:20:36.881169 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.879911    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/436fdcf3-5cd3-44e0-b5b8-3b878c58f6c7-iptables-alerter-script\") pod \"iptables-alerter-thqt6\" (UID: \"436fdcf3-5cd3-44e0-b5b8-3b878c58f6c7\") " pod="openshift-network-operator/iptables-alerter-thqt6"
Sep 09 15:20:36.881169 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.879937    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/587d946b-e46a-4e99-84bf-8ef93be089c9-etc-tuned\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.881169 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.879962    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4a532bf5-55b6-46a3-8c3f-4f3af4920a8d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cmk65\" (UID: \"4a532bf5-55b6-46a3-8c3f-4f3af4920a8d\") " pod="openshift-multus/multus-additional-cni-plugins-cmk65"
Sep 09 15:20:36.881169 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.879986    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-systemd-units\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.881169 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.879994    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/587d946b-e46a-4e99-84bf-8ef93be089c9-etc-kubernetes\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.881169 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.880018    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-log-socket\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.881169 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.880044    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/26591625-285a-4c93-b5be-7d0f4f0a0aae-dbus\") pod \"global-pull-secret-syncer-kcl4l\" (UID: \"26591625-285a-4c93-b5be-7d0f4f0a0aae\") " pod="kube-system/global-pull-secret-syncer-kcl4l"
Sep 09 15:20:36.881169 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.880067    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b60f4dd4-700c-4674-83ea-65b3d382f104-env-overrides\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.881169 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.880110    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b60f4dd4-700c-4674-83ea-65b3d382f104-ovn-node-metrics-cert\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.881169 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.880137    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/587d946b-e46a-4e99-84bf-8ef93be089c9-etc-sysconfig\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.881169 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.880172    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3a444239-19ad-490a-a006-0a43ce90c9ef-konnectivity-ca\") pod \"konnectivity-agent-xjfqb\" (UID: \"3a444239-19ad-490a-a006-0a43ce90c9ef\") " pod="kube-system/konnectivity-agent-xjfqb"
Sep 09 15:20:36.881169 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.880198    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a532bf5-55b6-46a3-8c3f-4f3af4920a8d-system-cni-dir\") pod \"multus-additional-cni-plugins-cmk65\" (UID: \"4a532bf5-55b6-46a3-8c3f-4f3af4920a8d\") " pod="openshift-multus/multus-additional-cni-plugins-cmk65"
Sep 09 15:20:36.881169 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.880213    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/587d946b-e46a-4e99-84bf-8ef93be089c9-etc-sysconfig\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.881169 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.880217    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-host-slash\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.881696 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.880275    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-var-lib-openvswitch\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.881696 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.880302    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-cnibin\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.881696 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.880328    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/587d946b-e46a-4e99-84bf-8ef93be089c9-etc-sysctl-conf\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.881696 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.880352    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/587d946b-e46a-4e99-84bf-8ef93be089c9-host\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.881696 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.880395    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-host-run-multus-certs\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.881696 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.880451    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/587d946b-e46a-4e99-84bf-8ef93be089c9-etc-sysctl-conf\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.881696 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.880458    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/587d946b-e46a-4e99-84bf-8ef93be089c9-host\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.881696 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.880491    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/587d946b-e46a-4e99-84bf-8ef93be089c9-etc-sysctl-d\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.881696 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.880519    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1aaf71f6-9195-4b50-a9a4-3cb85b10956a-host\") pod \"node-ca-txxcj\" (UID: \"1aaf71f6-9195-4b50-a9a4-3cb85b10956a\") " pod="openshift-image-registry/node-ca-txxcj"
Sep 09 15:20:36.881696 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.880543    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4a532bf5-55b6-46a3-8c3f-4f3af4920a8d-cni-binary-copy\") pod \"multus-additional-cni-plugins-cmk65\" (UID: \"4a532bf5-55b6-46a3-8c3f-4f3af4920a8d\") " pod="openshift-multus/multus-additional-cni-plugins-cmk65"
Sep 09 15:20:36.881696 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.880595    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1aaf71f6-9195-4b50-a9a4-3cb85b10956a-host\") pod \"node-ca-txxcj\" (UID: \"1aaf71f6-9195-4b50-a9a4-3cb85b10956a\") " pod="openshift-image-registry/node-ca-txxcj"
Sep 09 15:20:36.881696 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.880654    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/587d946b-e46a-4e99-84bf-8ef93be089c9-etc-sysctl-d\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.883128 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.883111    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/587d946b-e46a-4e99-84bf-8ef93be089c9-etc-tuned\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.883692 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.883677    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/587d946b-e46a-4e99-84bf-8ef93be089c9-tmp\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.889568 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.889543    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzkvx\" (UniqueName: \"kubernetes.io/projected/587d946b-e46a-4e99-84bf-8ef93be089c9-kube-api-access-zzkvx\") pod \"tuned-fwsrs\" (UID: \"587d946b-e46a-4e99-84bf-8ef93be089c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:36.889659 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.889593    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgmjh\" (UniqueName: \"kubernetes.io/projected/1aaf71f6-9195-4b50-a9a4-3cb85b10956a-kube-api-access-tgmjh\") pod \"node-ca-txxcj\" (UID: \"1aaf71f6-9195-4b50-a9a4-3cb85b10956a\") " pod="openshift-image-registry/node-ca-txxcj"
Sep 09 15:20:36.963470 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.963418    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-170.ec2.internal" event={"ID":"f934fa7e403724f763f7c170a46e1bd9","Type":"ContainerStarted","Data":"f26ba2ba0848ad0da60f13348f3976e56a39ee57e9a2722d11a5f05b880d009d"}
Sep 09 15:20:36.980936 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.980891    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-cni-binary-copy\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.980936 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.980939    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdtd6\" (UniqueName: \"kubernetes.io/projected/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-kube-api-access-bdtd6\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.981195 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.980966    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/436fdcf3-5cd3-44e0-b5b8-3b878c58f6c7-host-slash\") pod \"iptables-alerter-thqt6\" (UID: \"436fdcf3-5cd3-44e0-b5b8-3b878c58f6c7\") " pod="openshift-network-operator/iptables-alerter-thqt6"
Sep 09 15:20:36.981195 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.980992    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rjkr\" (UniqueName: \"kubernetes.io/projected/4a532bf5-55b6-46a3-8c3f-4f3af4920a8d-kube-api-access-8rjkr\") pod \"multus-additional-cni-plugins-cmk65\" (UID: \"4a532bf5-55b6-46a3-8c3f-4f3af4920a8d\") " pod="openshift-multus/multus-additional-cni-plugins-cmk65"
Sep 09 15:20:36.981195 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981092    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/436fdcf3-5cd3-44e0-b5b8-3b878c58f6c7-host-slash\") pod \"iptables-alerter-thqt6\" (UID: \"436fdcf3-5cd3-44e0-b5b8-3b878c58f6c7\") " pod="openshift-network-operator/iptables-alerter-thqt6"
Sep 09 15:20:36.981195 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981156    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-etc-openvswitch\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.981195 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981194    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-host-cni-bin\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.981533 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981219    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-multus-cni-dir\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.981533 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981235    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-etc-openvswitch\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.981533 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981245    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4a532bf5-55b6-46a3-8c3f-4f3af4920a8d-cnibin\") pod \"multus-additional-cni-plugins-cmk65\" (UID: \"4a532bf5-55b6-46a3-8c3f-4f3af4920a8d\") " pod="openshift-multus/multus-additional-cni-plugins-cmk65"
Sep 09 15:20:36.981533 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981273    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d957458f-a014-49cb-9403-efba9f50e7f9-socket-dir\") pod \"aws-ebs-csi-driver-node-xhn2q\" (UID: \"d957458f-a014-49cb-9403-efba9f50e7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhn2q"
Sep 09 15:20:36.981533 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981301    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-host-cni-bin\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.981533 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981308    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vgvr\" (UniqueName: \"kubernetes.io/projected/b60f4dd4-700c-4674-83ea-65b3d382f104-kube-api-access-4vgvr\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.981533 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981337    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4a532bf5-55b6-46a3-8c3f-4f3af4920a8d-cnibin\") pod \"multus-additional-cni-plugins-cmk65\" (UID: \"4a532bf5-55b6-46a3-8c3f-4f3af4920a8d\") " pod="openshift-multus/multus-additional-cni-plugins-cmk65"
Sep 09 15:20:36.981533 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981373    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26591625-285a-4c93-b5be-7d0f4f0a0aae-original-pull-secret\") pod \"global-pull-secret-syncer-kcl4l\" (UID: \"26591625-285a-4c93-b5be-7d0f4f0a0aae\") " pod="kube-system/global-pull-secret-syncer-kcl4l"
Sep 09 15:20:36.981533 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981405    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll8rs\" (UniqueName: \"kubernetes.io/projected/d830519d-a525-4029-ae78-d30bb04e19e5-kube-api-access-ll8rs\") pod \"network-check-target-xwx7m\" (UID: \"d830519d-a525-4029-ae78-d30bb04e19e5\") " pod="openshift-network-diagnostics/network-check-target-xwx7m"
Sep 09 15:20:36.981533 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981415    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-multus-cni-dir\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.981533 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981434    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-host-run-netns\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.981533 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981453    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d957458f-a014-49cb-9403-efba9f50e7f9-socket-dir\") pod \"aws-ebs-csi-driver-node-xhn2q\" (UID: \"d957458f-a014-49cb-9403-efba9f50e7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhn2q"
Sep 09 15:20:36.981533 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:36.981466    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 15:20:36.981533 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981474    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-host-run-netns\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.981533 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981517    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-multus-conf-dir\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.981533 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:36.981546    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26591625-285a-4c93-b5be-7d0f4f0a0aae-original-pull-secret podName:26591625-285a-4c93-b5be-7d0f4f0a0aae nodeName:}" failed. No retries permitted until 2026-09-09 15:20:37.481526232 +0000 UTC m=+3.080420856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/26591625-285a-4c93-b5be-7d0f4f0a0aae-original-pull-secret") pod "global-pull-secret-syncer-kcl4l" (UID: "26591625-285a-4c93-b5be-7d0f4f0a0aae") : object "kube-system"/"original-pull-secret" not registered
Sep 09 15:20:36.982387 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981464    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-multus-conf-dir\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.982387 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981583    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-host-cni-netd\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.982387 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981607    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b60f4dd4-700c-4674-83ea-65b3d382f104-ovnkube-config\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.982387 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981632    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cfc8b11-7c62-46fc-8a5e-d5d889501cfd-tmp-dir\") pod \"node-resolver-wgm9j\" (UID: \"3cfc8b11-7c62-46fc-8a5e-d5d889501cfd\") " pod="openshift-dns/node-resolver-wgm9j"
Sep 09 15:20:36.982387 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981644    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-host-cni-netd\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.982387 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981657    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-hostroot\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.982387 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981570    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-cni-binary-copy\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.982387 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981684    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4a532bf5-55b6-46a3-8c3f-4f3af4920a8d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cmk65\" (UID: \"4a532bf5-55b6-46a3-8c3f-4f3af4920a8d\") " pod="openshift-multus/multus-additional-cni-plugins-cmk65"
Sep 09 15:20:36.982387 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981711    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-host-run-netns\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.982387 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981714    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-hostroot\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.982387 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981735    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-run-systemd\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.982387 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981774    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d957458f-a014-49cb-9403-efba9f50e7f9-sys-fs\") pod \"aws-ebs-csi-driver-node-xhn2q\" (UID: \"d957458f-a014-49cb-9403-efba9f50e7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhn2q"
Sep 09 15:20:36.982387 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981801    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-run-openvswitch\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.982387 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981801    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-host-run-netns\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.982387 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981820    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-run-systemd\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.982387 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981825    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4a532bf5-55b6-46a3-8c3f-4f3af4920a8d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cmk65\" (UID: \"4a532bf5-55b6-46a3-8c3f-4f3af4920a8d\") " pod="openshift-multus/multus-additional-cni-plugins-cmk65"
Sep 09 15:20:36.982387 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981865    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d957458f-a014-49cb-9403-efba9f50e7f9-sys-fs\") pod \"aws-ebs-csi-driver-node-xhn2q\" (UID: \"d957458f-a014-49cb-9403-efba9f50e7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhn2q"
Sep 09 15:20:36.982387 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981894    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57473bc0-f6d0-45f1-8954-93a7d2f4321b-metrics-certs\") pod \"network-metrics-daemon-v5kbj\" (UID: \"57473bc0-f6d0-45f1-8954-93a7d2f4321b\") " pod="openshift-multus/network-metrics-daemon-v5kbj"
Sep 09 15:20:36.983358 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981905    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4a532bf5-55b6-46a3-8c3f-4f3af4920a8d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cmk65\" (UID: \"4a532bf5-55b6-46a3-8c3f-4f3af4920a8d\") " pod="openshift-multus/multus-additional-cni-plugins-cmk65"
Sep 09 15:20:36.983358 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981924    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-system-cni-dir\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.983358 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981954    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-host-var-lib-cni-bin\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.983358 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981985    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-etc-kubernetes\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.983358 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:36.982008    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:20:36.983358 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.982013    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/26591625-285a-4c93-b5be-7d0f4f0a0aae-kubelet-config\") pod \"global-pull-secret-syncer-kcl4l\" (UID: \"26591625-285a-4c93-b5be-7d0f4f0a0aae\") " pod="kube-system/global-pull-secret-syncer-kcl4l"
Sep 09 15:20:36.983358 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.981861    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-run-openvswitch\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.983358 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.982044    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-run-ovn\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.983358 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:36.982073    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57473bc0-f6d0-45f1-8954-93a7d2f4321b-metrics-certs podName:57473bc0-f6d0-45f1-8954-93a7d2f4321b nodeName:}" failed. No retries permitted until 2026-09-09 15:20:37.482055378 +0000 UTC m=+3.080950015 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57473bc0-f6d0-45f1-8954-93a7d2f4321b-metrics-certs") pod "network-metrics-daemon-v5kbj" (UID: "57473bc0-f6d0-45f1-8954-93a7d2f4321b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:20:36.983358 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.982082    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-host-var-lib-cni-bin\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.983358 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.982101    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-host-run-ovn-kubernetes\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.983358 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.982109    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-run-ovn\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.983358 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.982008    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cfc8b11-7c62-46fc-8a5e-d5d889501cfd-tmp-dir\") pod \"node-resolver-wgm9j\" (UID: \"3cfc8b11-7c62-46fc-8a5e-d5d889501cfd\") " pod="openshift-dns/node-resolver-wgm9j"
Sep 09 15:20:36.983358 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.982148    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-system-cni-dir\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.983358 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.982153    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-host-run-ovn-kubernetes\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.983358 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.982186    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b60f4dd4-700c-4674-83ea-65b3d382f104-ovnkube-script-lib\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.983358 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.982196    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/26591625-285a-4c93-b5be-7d0f4f0a0aae-kubelet-config\") pod \"global-pull-secret-syncer-kcl4l\" (UID: \"26591625-285a-4c93-b5be-7d0f4f0a0aae\") " pod="kube-system/global-pull-secret-syncer-kcl4l"
Sep 09 15:20:36.984098 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.982208    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-etc-kubernetes\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.984098 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.982224    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-multus-socket-dir-parent\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.984098 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.982250    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-host-run-k8s-cni-cncf-io\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.984098 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.982294    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-host-run-k8s-cni-cncf-io\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.984098 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.982299    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b60f4dd4-700c-4674-83ea-65b3d382f104-ovnkube-config\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.984098 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.982302    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-multus-socket-dir-parent\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.984098 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.982321    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvznv\" (UniqueName: \"kubernetes.io/projected/436fdcf3-5cd3-44e0-b5b8-3b878c58f6c7-kube-api-access-lvznv\") pod \"iptables-alerter-thqt6\" (UID: \"436fdcf3-5cd3-44e0-b5b8-3b878c58f6c7\") " pod="openshift-network-operator/iptables-alerter-thqt6"
Sep 09 15:20:36.984098 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.982334    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4a532bf5-55b6-46a3-8c3f-4f3af4920a8d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cmk65\" (UID: \"4a532bf5-55b6-46a3-8c3f-4f3af4920a8d\") " pod="openshift-multus/multus-additional-cni-plugins-cmk65"
Sep 09 15:20:36.984098 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.982364    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3a444239-19ad-490a-a006-0a43ce90c9ef-agent-certs\") pod \"konnectivity-agent-xjfqb\" (UID: \"3a444239-19ad-490a-a006-0a43ce90c9ef\") " pod="kube-system/konnectivity-agent-xjfqb"
Sep 09 15:20:36.984098 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.982393    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-node-log\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.984098 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.982440    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-node-log\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.984098 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.982512    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.984098 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.982549    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzd95\" (UniqueName: \"kubernetes.io/projected/3cfc8b11-7c62-46fc-8a5e-d5d889501cfd-kube-api-access-bzd95\") pod \"node-resolver-wgm9j\" (UID: \"3cfc8b11-7c62-46fc-8a5e-d5d889501cfd\") " pod="openshift-dns/node-resolver-wgm9j"
Sep 09 15:20:36.984098 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.982558    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.984098 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.982586    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7sx8\" (UniqueName: \"kubernetes.io/projected/d957458f-a014-49cb-9403-efba9f50e7f9-kube-api-access-x7sx8\") pod \"aws-ebs-csi-driver-node-xhn2q\" (UID: \"d957458f-a014-49cb-9403-efba9f50e7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhn2q"
Sep 09 15:20:36.984098 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.982614    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-multus-daemon-config\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.984098 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.982641    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d957458f-a014-49cb-9403-efba9f50e7f9-registration-dir\") pod \"aws-ebs-csi-driver-node-xhn2q\" (UID: \"d957458f-a014-49cb-9403-efba9f50e7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhn2q"
Sep 09 15:20:36.984600 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.982667    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnn5b\" (UniqueName: \"kubernetes.io/projected/57473bc0-f6d0-45f1-8954-93a7d2f4321b-kube-api-access-lnn5b\") pod \"network-metrics-daemon-v5kbj\" (UID: \"57473bc0-f6d0-45f1-8954-93a7d2f4321b\") " pod="openshift-multus/network-metrics-daemon-v5kbj"
Sep 09 15:20:36.984600 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.982669    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b60f4dd4-700c-4674-83ea-65b3d382f104-ovnkube-script-lib\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.984600 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.982691    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/436fdcf3-5cd3-44e0-b5b8-3b878c58f6c7-iptables-alerter-script\") pod \"iptables-alerter-thqt6\" (UID: \"436fdcf3-5cd3-44e0-b5b8-3b878c58f6c7\") " pod="openshift-network-operator/iptables-alerter-thqt6"
Sep 09 15:20:36.984600 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.982740    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d957458f-a014-49cb-9403-efba9f50e7f9-registration-dir\") pod \"aws-ebs-csi-driver-node-xhn2q\" (UID: \"d957458f-a014-49cb-9403-efba9f50e7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhn2q"
Sep 09 15:20:36.984600 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.982748    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4a532bf5-55b6-46a3-8c3f-4f3af4920a8d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cmk65\" (UID: \"4a532bf5-55b6-46a3-8c3f-4f3af4920a8d\") " pod="openshift-multus/multus-additional-cni-plugins-cmk65"
Sep 09 15:20:36.984600 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.982856    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-systemd-units\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.984600 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.982886    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-log-socket\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.984600 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.982900    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-systemd-units\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.984600 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.982912    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/26591625-285a-4c93-b5be-7d0f4f0a0aae-dbus\") pod \"global-pull-secret-syncer-kcl4l\" (UID: \"26591625-285a-4c93-b5be-7d0f4f0a0aae\") " pod="kube-system/global-pull-secret-syncer-kcl4l"
Sep 09 15:20:36.984600 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.982950    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-log-socket\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.984600 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.982973    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/26591625-285a-4c93-b5be-7d0f4f0a0aae-dbus\") pod \"global-pull-secret-syncer-kcl4l\" (UID: \"26591625-285a-4c93-b5be-7d0f4f0a0aae\") " pod="kube-system/global-pull-secret-syncer-kcl4l"
Sep 09 15:20:36.984600 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.982979    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b60f4dd4-700c-4674-83ea-65b3d382f104-env-overrides\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.984600 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.983011    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b60f4dd4-700c-4674-83ea-65b3d382f104-ovn-node-metrics-cert\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.984600 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.983032    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3a444239-19ad-490a-a006-0a43ce90c9ef-konnectivity-ca\") pod \"konnectivity-agent-xjfqb\" (UID: \"3a444239-19ad-490a-a006-0a43ce90c9ef\") " pod="kube-system/konnectivity-agent-xjfqb"
Sep 09 15:20:36.984600 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.983053    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a532bf5-55b6-46a3-8c3f-4f3af4920a8d-system-cni-dir\") pod \"multus-additional-cni-plugins-cmk65\" (UID: \"4a532bf5-55b6-46a3-8c3f-4f3af4920a8d\") " pod="openshift-multus/multus-additional-cni-plugins-cmk65"
Sep 09 15:20:36.984600 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.983079    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-host-slash\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.984600 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.983104    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-var-lib-openvswitch\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.985158 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.983128    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-cnibin\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.985158 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.983157    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-host-run-multus-certs\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.985158 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.983185    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4a532bf5-55b6-46a3-8c3f-4f3af4920a8d-cni-binary-copy\") pod \"multus-additional-cni-plugins-cmk65\" (UID: \"4a532bf5-55b6-46a3-8c3f-4f3af4920a8d\") " pod="openshift-multus/multus-additional-cni-plugins-cmk65"
Sep 09 15:20:36.985158 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.983210    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d957458f-a014-49cb-9403-efba9f50e7f9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xhn2q\" (UID: \"d957458f-a014-49cb-9403-efba9f50e7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhn2q"
Sep 09 15:20:36.985158 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.983209    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-host-slash\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.985158 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.983273    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-var-lib-openvswitch\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.985158 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.983327    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-cnibin\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.985158 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.983370    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-host-run-multus-certs\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.985158 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.983417    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b60f4dd4-700c-4674-83ea-65b3d382f104-env-overrides\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.985158 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.983461    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/436fdcf3-5cd3-44e0-b5b8-3b878c58f6c7-iptables-alerter-script\") pod \"iptables-alerter-thqt6\" (UID: \"436fdcf3-5cd3-44e0-b5b8-3b878c58f6c7\") " pod="openshift-network-operator/iptables-alerter-thqt6"
Sep 09 15:20:36.985158 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.983484    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a532bf5-55b6-46a3-8c3f-4f3af4920a8d-system-cni-dir\") pod \"multus-additional-cni-plugins-cmk65\" (UID: \"4a532bf5-55b6-46a3-8c3f-4f3af4920a8d\") " pod="openshift-multus/multus-additional-cni-plugins-cmk65"
Sep 09 15:20:36.985158 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.983534    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4a532bf5-55b6-46a3-8c3f-4f3af4920a8d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cmk65\" (UID: \"4a532bf5-55b6-46a3-8c3f-4f3af4920a8d\") " pod="openshift-multus/multus-additional-cni-plugins-cmk65"
Sep 09 15:20:36.985158 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.983547    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d957458f-a014-49cb-9403-efba9f50e7f9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xhn2q\" (UID: \"d957458f-a014-49cb-9403-efba9f50e7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhn2q"
Sep 09 15:20:36.985158 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.983586    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d957458f-a014-49cb-9403-efba9f50e7f9-device-dir\") pod \"aws-ebs-csi-driver-node-xhn2q\" (UID: \"d957458f-a014-49cb-9403-efba9f50e7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhn2q"
Sep 09 15:20:36.985158 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.983620    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d957458f-a014-49cb-9403-efba9f50e7f9-etc-selinux\") pod \"aws-ebs-csi-driver-node-xhn2q\" (UID: \"d957458f-a014-49cb-9403-efba9f50e7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhn2q"
Sep 09 15:20:36.985158 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.983648    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-host-var-lib-cni-multus\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.985158 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.983675    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-host-var-lib-kubelet\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.985659 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.983697    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-host-kubelet\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.985659 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.983698    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d957458f-a014-49cb-9403-efba9f50e7f9-device-dir\") pod \"aws-ebs-csi-driver-node-xhn2q\" (UID: \"d957458f-a014-49cb-9403-efba9f50e7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhn2q"
Sep 09 15:20:36.985659 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.983742    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b60f4dd4-700c-4674-83ea-65b3d382f104-host-kubelet\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.985659 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.983753    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3cfc8b11-7c62-46fc-8a5e-d5d889501cfd-hosts-file\") pod \"node-resolver-wgm9j\" (UID: \"3cfc8b11-7c62-46fc-8a5e-d5d889501cfd\") " pod="openshift-dns/node-resolver-wgm9j"
Sep 09 15:20:36.985659 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.983764    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3a444239-19ad-490a-a006-0a43ce90c9ef-konnectivity-ca\") pod \"konnectivity-agent-xjfqb\" (UID: \"3a444239-19ad-490a-a006-0a43ce90c9ef\") " pod="kube-system/konnectivity-agent-xjfqb"
Sep 09 15:20:36.985659 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.983771    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d957458f-a014-49cb-9403-efba9f50e7f9-etc-selinux\") pod \"aws-ebs-csi-driver-node-xhn2q\" (UID: \"d957458f-a014-49cb-9403-efba9f50e7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhn2q"
Sep 09 15:20:36.985659 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.983785    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-host-var-lib-cni-multus\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.985659 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.983830    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-host-var-lib-kubelet\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.985659 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.983838    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3cfc8b11-7c62-46fc-8a5e-d5d889501cfd-hosts-file\") pod \"node-resolver-wgm9j\" (UID: \"3cfc8b11-7c62-46fc-8a5e-d5d889501cfd\") " pod="openshift-dns/node-resolver-wgm9j"
Sep 09 15:20:36.985659 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.983892    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4a532bf5-55b6-46a3-8c3f-4f3af4920a8d-cni-binary-copy\") pod \"multus-additional-cni-plugins-cmk65\" (UID: \"4a532bf5-55b6-46a3-8c3f-4f3af4920a8d\") " pod="openshift-multus/multus-additional-cni-plugins-cmk65"
Sep 09 15:20:36.985659 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.983950    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-multus-daemon-config\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:36.985659 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.985068    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3a444239-19ad-490a-a006-0a43ce90c9ef-agent-certs\") pod \"konnectivity-agent-xjfqb\" (UID: \"3a444239-19ad-490a-a006-0a43ce90c9ef\") " pod="kube-system/konnectivity-agent-xjfqb"
Sep 09 15:20:36.985659 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:36.985325    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b60f4dd4-700c-4674-83ea-65b3d382f104-ovn-node-metrics-cert\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:36.999705 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:36.999525    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 15:20:36.999705 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:36.999551    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 15:20:36.999705 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:36.999567    2553 projected.go:196] Error preparing data for projected volume kube-api-access-ll8rs for pod openshift-network-diagnostics/network-check-target-xwx7m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:20:36.999705 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:36.999679    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d830519d-a525-4029-ae78-d30bb04e19e5-kube-api-access-ll8rs podName:d830519d-a525-4029-ae78-d30bb04e19e5 nodeName:}" failed. No retries permitted until 2026-09-09 15:20:37.499616331 +0000 UTC m=+3.098510972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ll8rs" (UniqueName: "kubernetes.io/projected/d830519d-a525-4029-ae78-d30bb04e19e5-kube-api-access-ll8rs") pod "network-check-target-xwx7m" (UID: "d830519d-a525-4029-ae78-d30bb04e19e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:20:37.001352 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:37.001318    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzd95\" (UniqueName: \"kubernetes.io/projected/3cfc8b11-7c62-46fc-8a5e-d5d889501cfd-kube-api-access-bzd95\") pod \"node-resolver-wgm9j\" (UID: \"3cfc8b11-7c62-46fc-8a5e-d5d889501cfd\") " pod="openshift-dns/node-resolver-wgm9j"
Sep 09 15:20:37.001622 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:37.001592    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvznv\" (UniqueName: \"kubernetes.io/projected/436fdcf3-5cd3-44e0-b5b8-3b878c58f6c7-kube-api-access-lvznv\") pod \"iptables-alerter-thqt6\" (UID: \"436fdcf3-5cd3-44e0-b5b8-3b878c58f6c7\") " pod="openshift-network-operator/iptables-alerter-thqt6"
Sep 09 15:20:37.001790 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:37.001755    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7sx8\" (UniqueName: \"kubernetes.io/projected/d957458f-a014-49cb-9403-efba9f50e7f9-kube-api-access-x7sx8\") pod \"aws-ebs-csi-driver-node-xhn2q\" (UID: \"d957458f-a014-49cb-9403-efba9f50e7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhn2q"
Sep 09 15:20:37.001870 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:37.001827    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdtd6\" (UniqueName: \"kubernetes.io/projected/d1a7a818-24f1-4d7c-a3d5-416915c6c2cf-kube-api-access-bdtd6\") pod \"multus-bt4f2\" (UID: \"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf\") " pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:37.002223 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:37.002206    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rjkr\" (UniqueName: \"kubernetes.io/projected/4a532bf5-55b6-46a3-8c3f-4f3af4920a8d-kube-api-access-8rjkr\") pod \"multus-additional-cni-plugins-cmk65\" (UID: \"4a532bf5-55b6-46a3-8c3f-4f3af4920a8d\") " pod="openshift-multus/multus-additional-cni-plugins-cmk65"
Sep 09 15:20:37.002315 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:37.002293    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnn5b\" (UniqueName: \"kubernetes.io/projected/57473bc0-f6d0-45f1-8954-93a7d2f4321b-kube-api-access-lnn5b\") pod \"network-metrics-daemon-v5kbj\" (UID: \"57473bc0-f6d0-45f1-8954-93a7d2f4321b\") " pod="openshift-multus/network-metrics-daemon-v5kbj"
Sep 09 15:20:37.003363 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:37.003343    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vgvr\" (UniqueName: \"kubernetes.io/projected/b60f4dd4-700c-4674-83ea-65b3d382f104-kube-api-access-4vgvr\") pod \"ovnkube-node-45pwd\" (UID: \"b60f4dd4-700c-4674-83ea-65b3d382f104\") " pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:37.023629 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:37.023612    2553 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 15:20:37.059504 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:37.059460    2553 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 15:20:37.075038 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:37.075004    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fwsrs"
Sep 09 15:20:37.083627 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:37.081194    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-txxcj"
Sep 09 15:20:37.087870 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:37.087842    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod587d946b_e46a_4e99_84bf_8ef93be089c9.slice/crio-d2f5aa18546e2654c4be8e0d5819bee95eeeb5a85f4042c7556979006637319b WatchSource:0}: Error finding container d2f5aa18546e2654c4be8e0d5819bee95eeeb5a85f4042c7556979006637319b: Status 404 returned error can't find the container with id d2f5aa18546e2654c4be8e0d5819bee95eeeb5a85f4042c7556979006637319b
Sep 09 15:20:37.089111 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:37.089086    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xjfqb"
Sep 09 15:20:37.095203 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:37.095177    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aaf71f6_9195_4b50_a9a4_3cb85b10956a.slice/crio-1eaf525dd3d554b551ca52b016f267ac8b64c7c5784e7392e08c49d677fdcb6a WatchSource:0}: Error finding container 1eaf525dd3d554b551ca52b016f267ac8b64c7c5784e7392e08c49d677fdcb6a: Status 404 returned error can't find the container with id 1eaf525dd3d554b551ca52b016f267ac8b64c7c5784e7392e08c49d677fdcb6a
Sep 09 15:20:37.097998 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:37.097803    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhn2q"
Sep 09 15:20:37.101245 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:37.100832    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a444239_19ad_490a_a006_0a43ce90c9ef.slice/crio-db0786529acca76f994c2a63635cfc82ddd7105017dbc11af60633160e26c279 WatchSource:0}: Error finding container db0786529acca76f994c2a63635cfc82ddd7105017dbc11af60633160e26c279: Status 404 returned error can't find the container with id db0786529acca76f994c2a63635cfc82ddd7105017dbc11af60633160e26c279
Sep 09 15:20:37.103352 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:37.103330    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cmk65"
Sep 09 15:20:37.109364 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:37.108906    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:20:37.110382 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:37.110349    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd957458f_a014_49cb_9403_efba9f50e7f9.slice/crio-a3cda8229aade019ed3cd77e1b14dba70d4c0bd30a574d6acd4eddb47cae880d WatchSource:0}: Error finding container a3cda8229aade019ed3cd77e1b14dba70d4c0bd30a574d6acd4eddb47cae880d: Status 404 returned error can't find the container with id a3cda8229aade019ed3cd77e1b14dba70d4c0bd30a574d6acd4eddb47cae880d
Sep 09 15:20:37.115125 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:37.115100    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bt4f2"
Sep 09 15:20:37.119736 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:37.119711    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a532bf5_55b6_46a3_8c3f_4f3af4920a8d.slice/crio-59f04119f2c98fe8314c954c3ff4318e3c88de39f07a41f8f33819b2e58821bc WatchSource:0}: Error finding container 59f04119f2c98fe8314c954c3ff4318e3c88de39f07a41f8f33819b2e58821bc: Status 404 returned error can't find the container with id 59f04119f2c98fe8314c954c3ff4318e3c88de39f07a41f8f33819b2e58821bc
Sep 09 15:20:37.123803 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:37.123778    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb60f4dd4_700c_4674_83ea_65b3d382f104.slice/crio-0c9b1a3ec051677bef32f1fc586d2889054daa94261dfc0dc4621eb5d3caf58d WatchSource:0}: Error finding container 0c9b1a3ec051677bef32f1fc586d2889054daa94261dfc0dc4621eb5d3caf58d: Status 404 returned error can't find the container with id 0c9b1a3ec051677bef32f1fc586d2889054daa94261dfc0dc4621eb5d3caf58d
Sep 09 15:20:37.124801 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:37.124639    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-thqt6"
Sep 09 15:20:37.126407 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:37.126382    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1a7a818_24f1_4d7c_a3d5_416915c6c2cf.slice/crio-b0b771bfb64ead010d619f92a91a7839ad66f868f4776b111c3730b19d98046d WatchSource:0}: Error finding container b0b771bfb64ead010d619f92a91a7839ad66f868f4776b111c3730b19d98046d: Status 404 returned error can't find the container with id b0b771bfb64ead010d619f92a91a7839ad66f868f4776b111c3730b19d98046d
Sep 09 15:20:37.130179 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:37.129963    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wgm9j"
Sep 09 15:20:37.138563 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:37.138536    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod436fdcf3_5cd3_44e0_b5b8_3b878c58f6c7.slice/crio-1bc15f548a9f88541eb694bd0a8c861689ef17934941e20fce9dbe286979dd94 WatchSource:0}: Error finding container 1bc15f548a9f88541eb694bd0a8c861689ef17934941e20fce9dbe286979dd94: Status 404 returned error can't find the container with id 1bc15f548a9f88541eb694bd0a8c861689ef17934941e20fce9dbe286979dd94
Sep 09 15:20:37.142524 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:37.142502    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cfc8b11_7c62_46fc_8a5e_d5d889501cfd.slice/crio-bdbe4d9c6481e7d66ea2085e986f67ba58739e5ebb6da4d9688331dfd284f44d WatchSource:0}: Error finding container bdbe4d9c6481e7d66ea2085e986f67ba58739e5ebb6da4d9688331dfd284f44d: Status 404 returned error can't find the container with id bdbe4d9c6481e7d66ea2085e986f67ba58739e5ebb6da4d9688331dfd284f44d
Sep 09 15:20:37.487540 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:37.487434    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57473bc0-f6d0-45f1-8954-93a7d2f4321b-metrics-certs\") pod \"network-metrics-daemon-v5kbj\" (UID: \"57473bc0-f6d0-45f1-8954-93a7d2f4321b\") " pod="openshift-multus/network-metrics-daemon-v5kbj"
Sep 09 15:20:37.487710 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:37.487577    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26591625-285a-4c93-b5be-7d0f4f0a0aae-original-pull-secret\") pod \"global-pull-secret-syncer-kcl4l\" (UID: \"26591625-285a-4c93-b5be-7d0f4f0a0aae\") " pod="kube-system/global-pull-secret-syncer-kcl4l"
Sep 09 15:20:37.487710 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:37.487605    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:20:37.487710 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:37.487701    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57473bc0-f6d0-45f1-8954-93a7d2f4321b-metrics-certs podName:57473bc0-f6d0-45f1-8954-93a7d2f4321b nodeName:}" failed. No retries permitted until 2026-09-09 15:20:38.487663074 +0000 UTC m=+4.086557719 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57473bc0-f6d0-45f1-8954-93a7d2f4321b-metrics-certs") pod "network-metrics-daemon-v5kbj" (UID: "57473bc0-f6d0-45f1-8954-93a7d2f4321b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:20:37.488008 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:37.487748    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 15:20:37.488008 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:37.487899    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26591625-285a-4c93-b5be-7d0f4f0a0aae-original-pull-secret podName:26591625-285a-4c93-b5be-7d0f4f0a0aae nodeName:}" failed. No retries permitted until 2026-09-09 15:20:38.487874744 +0000 UTC m=+4.086769370 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/26591625-285a-4c93-b5be-7d0f4f0a0aae-original-pull-secret") pod "global-pull-secret-syncer-kcl4l" (UID: "26591625-285a-4c93-b5be-7d0f4f0a0aae") : object "kube-system"/"original-pull-secret" not registered
Sep 09 15:20:37.588007 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:37.587973    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll8rs\" (UniqueName: \"kubernetes.io/projected/d830519d-a525-4029-ae78-d30bb04e19e5-kube-api-access-ll8rs\") pod \"network-check-target-xwx7m\" (UID: \"d830519d-a525-4029-ae78-d30bb04e19e5\") " pod="openshift-network-diagnostics/network-check-target-xwx7m"
Sep 09 15:20:37.588214 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:37.588146    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 15:20:37.588214 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:37.588167    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 15:20:37.588214 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:37.588179    2553 projected.go:196] Error preparing data for projected volume kube-api-access-ll8rs for pod openshift-network-diagnostics/network-check-target-xwx7m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:20:37.588386 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:37.588255    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d830519d-a525-4029-ae78-d30bb04e19e5-kube-api-access-ll8rs podName:d830519d-a525-4029-ae78-d30bb04e19e5 nodeName:}" failed. No retries permitted until 2026-09-09 15:20:38.588235695 +0000 UTC m=+4.187130339 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ll8rs" (UniqueName: "kubernetes.io/projected/d830519d-a525-4029-ae78-d30bb04e19e5-kube-api-access-ll8rs") pod "network-check-target-xwx7m" (UID: "d830519d-a525-4029-ae78-d30bb04e19e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:20:37.810666 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:37.810559    2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 15:15:37 +0000 UTC" deadline="2028-03-09 00:41:00.869070256 +0000 UTC"
Sep 09 15:20:37.810666 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:37.810622    2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13113h20m23.058473472s"
Sep 09 15:20:37.956128 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:37.956085    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwx7m"
Sep 09 15:20:37.956294 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:37.956235    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xwx7m" podUID="d830519d-a525-4029-ae78-d30bb04e19e5"
Sep 09 15:20:37.967003 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:37.966964    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cmk65" event={"ID":"4a532bf5-55b6-46a3-8c3f-4f3af4920a8d","Type":"ContainerStarted","Data":"59f04119f2c98fe8314c954c3ff4318e3c88de39f07a41f8f33819b2e58821bc"}
Sep 09 15:20:37.968273 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:37.968242    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhn2q" event={"ID":"d957458f-a014-49cb-9403-efba9f50e7f9","Type":"ContainerStarted","Data":"a3cda8229aade019ed3cd77e1b14dba70d4c0bd30a574d6acd4eddb47cae880d"}
Sep 09 15:20:37.969641 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:37.969617    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-txxcj" event={"ID":"1aaf71f6-9195-4b50-a9a4-3cb85b10956a","Type":"ContainerStarted","Data":"1eaf525dd3d554b551ca52b016f267ac8b64c7c5784e7392e08c49d677fdcb6a"}
Sep 09 15:20:37.970964 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:37.970936    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fwsrs" event={"ID":"587d946b-e46a-4e99-84bf-8ef93be089c9","Type":"ContainerStarted","Data":"d2f5aa18546e2654c4be8e0d5819bee95eeeb5a85f4042c7556979006637319b"}
Sep 09 15:20:37.972189 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:37.972161    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wgm9j" event={"ID":"3cfc8b11-7c62-46fc-8a5e-d5d889501cfd","Type":"ContainerStarted","Data":"bdbe4d9c6481e7d66ea2085e986f67ba58739e5ebb6da4d9688331dfd284f44d"}
Sep 09 15:20:37.973487 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:37.973463    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-thqt6" event={"ID":"436fdcf3-5cd3-44e0-b5b8-3b878c58f6c7","Type":"ContainerStarted","Data":"1bc15f548a9f88541eb694bd0a8c861689ef17934941e20fce9dbe286979dd94"}
Sep 09 15:20:37.974865 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:37.974843    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bt4f2" event={"ID":"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf","Type":"ContainerStarted","Data":"b0b771bfb64ead010d619f92a91a7839ad66f868f4776b111c3730b19d98046d"}
Sep 09 15:20:37.976259 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:37.976231    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-45pwd" event={"ID":"b60f4dd4-700c-4674-83ea-65b3d382f104","Type":"ContainerStarted","Data":"0c9b1a3ec051677bef32f1fc586d2889054daa94261dfc0dc4621eb5d3caf58d"}
Sep 09 15:20:37.977619 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:37.977593    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xjfqb" event={"ID":"3a444239-19ad-490a-a006-0a43ce90c9ef","Type":"ContainerStarted","Data":"db0786529acca76f994c2a63635cfc82ddd7105017dbc11af60633160e26c279"}
Sep 09 15:20:38.496201 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:38.496097    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26591625-285a-4c93-b5be-7d0f4f0a0aae-original-pull-secret\") pod \"global-pull-secret-syncer-kcl4l\" (UID: \"26591625-285a-4c93-b5be-7d0f4f0a0aae\") " pod="kube-system/global-pull-secret-syncer-kcl4l"
Sep 09 15:20:38.496201 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:38.496179    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57473bc0-f6d0-45f1-8954-93a7d2f4321b-metrics-certs\") pod \"network-metrics-daemon-v5kbj\" (UID: \"57473bc0-f6d0-45f1-8954-93a7d2f4321b\") " pod="openshift-multus/network-metrics-daemon-v5kbj"
Sep 09 15:20:38.496427 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:38.496361    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:20:38.496489 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:38.496432    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57473bc0-f6d0-45f1-8954-93a7d2f4321b-metrics-certs podName:57473bc0-f6d0-45f1-8954-93a7d2f4321b nodeName:}" failed. No retries permitted until 2026-09-09 15:20:40.4964113 +0000 UTC m=+6.095305942 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57473bc0-f6d0-45f1-8954-93a7d2f4321b-metrics-certs") pod "network-metrics-daemon-v5kbj" (UID: "57473bc0-f6d0-45f1-8954-93a7d2f4321b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:20:38.497022 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:38.496928    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 15:20:38.497022 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:38.496987    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26591625-285a-4c93-b5be-7d0f4f0a0aae-original-pull-secret podName:26591625-285a-4c93-b5be-7d0f4f0a0aae nodeName:}" failed. No retries permitted until 2026-09-09 15:20:40.496969884 +0000 UTC m=+6.095864510 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/26591625-285a-4c93-b5be-7d0f4f0a0aae-original-pull-secret") pod "global-pull-secret-syncer-kcl4l" (UID: "26591625-285a-4c93-b5be-7d0f4f0a0aae") : object "kube-system"/"original-pull-secret" not registered
Sep 09 15:20:38.598307 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:38.597519    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll8rs\" (UniqueName: \"kubernetes.io/projected/d830519d-a525-4029-ae78-d30bb04e19e5-kube-api-access-ll8rs\") pod \"network-check-target-xwx7m\" (UID: \"d830519d-a525-4029-ae78-d30bb04e19e5\") " pod="openshift-network-diagnostics/network-check-target-xwx7m"
Sep 09 15:20:38.598307 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:38.597765    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 15:20:38.598307 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:38.597809    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 15:20:38.598307 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:38.597825    2553 projected.go:196] Error preparing data for projected volume kube-api-access-ll8rs for pod openshift-network-diagnostics/network-check-target-xwx7m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:20:38.598307 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:38.597896    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d830519d-a525-4029-ae78-d30bb04e19e5-kube-api-access-ll8rs podName:d830519d-a525-4029-ae78-d30bb04e19e5 nodeName:}" failed. No retries permitted until 2026-09-09 15:20:40.597875819 +0000 UTC m=+6.196770466 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ll8rs" (UniqueName: "kubernetes.io/projected/d830519d-a525-4029-ae78-d30bb04e19e5-kube-api-access-ll8rs") pod "network-check-target-xwx7m" (UID: "d830519d-a525-4029-ae78-d30bb04e19e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:20:38.813596 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:38.811476    2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 15:15:37 +0000 UTC" deadline="2028-06-09 07:37:44.949345668 +0000 UTC"
Sep 09 15:20:38.813596 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:38.811546    2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15328h17m6.137804095s"
Sep 09 15:20:38.959329 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:38.959278    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5kbj"
Sep 09 15:20:38.959510 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:38.959422    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5kbj" podUID="57473bc0-f6d0-45f1-8954-93a7d2f4321b"
Sep 09 15:20:38.959868 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:38.959844    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcl4l"
Sep 09 15:20:38.960005 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:38.959954    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcl4l" podUID="26591625-285a-4c93-b5be-7d0f4f0a0aae"
Sep 09 15:20:38.995490 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:38.995446    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-170.ec2.internal" event={"ID":"f934fa7e403724f763f7c170a46e1bd9","Type":"ContainerStarted","Data":"251c5d9e9401385b93fbd01fb281a7e9e622a2c74e1afa45493f233302b9b111"}
Sep 09 15:20:39.066033 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:20:39.061639    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17b2b5583df839506650e89cca8d5eb6.slice/crio-d6631ecb22d4ac2ce82cf470c6dc47f26d1386a1cbd1b60253c701680d0a344a WatchSource:0}: Error finding container d6631ecb22d4ac2ce82cf470c6dc47f26d1386a1cbd1b60253c701680d0a344a: Status 404 returned error can't find the container with id d6631ecb22d4ac2ce82cf470c6dc47f26d1386a1cbd1b60253c701680d0a344a
Sep 09 15:20:39.956664 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:39.955995    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwx7m"
Sep 09 15:20:39.956664 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:39.956178    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xwx7m" podUID="d830519d-a525-4029-ae78-d30bb04e19e5"
Sep 09 15:20:40.001494 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:40.001109    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-170.ec2.internal" event={"ID":"17b2b5583df839506650e89cca8d5eb6","Type":"ContainerStarted","Data":"d6631ecb22d4ac2ce82cf470c6dc47f26d1386a1cbd1b60253c701680d0a344a"}
Sep 09 15:20:40.519180 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:40.519124    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26591625-285a-4c93-b5be-7d0f4f0a0aae-original-pull-secret\") pod \"global-pull-secret-syncer-kcl4l\" (UID: \"26591625-285a-4c93-b5be-7d0f4f0a0aae\") " pod="kube-system/global-pull-secret-syncer-kcl4l"
Sep 09 15:20:40.519392 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:40.519205    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57473bc0-f6d0-45f1-8954-93a7d2f4321b-metrics-certs\") pod \"network-metrics-daemon-v5kbj\" (UID: \"57473bc0-f6d0-45f1-8954-93a7d2f4321b\") " pod="openshift-multus/network-metrics-daemon-v5kbj"
Sep 09 15:20:40.519392 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:40.519334    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:20:40.519727 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:40.519401    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57473bc0-f6d0-45f1-8954-93a7d2f4321b-metrics-certs podName:57473bc0-f6d0-45f1-8954-93a7d2f4321b nodeName:}" failed. No retries permitted until 2026-09-09 15:20:44.519380828 +0000 UTC m=+10.118275452 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57473bc0-f6d0-45f1-8954-93a7d2f4321b-metrics-certs") pod "network-metrics-daemon-v5kbj" (UID: "57473bc0-f6d0-45f1-8954-93a7d2f4321b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:20:40.519902 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:40.519838    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 15:20:40.519902 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:40.519892    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26591625-285a-4c93-b5be-7d0f4f0a0aae-original-pull-secret podName:26591625-285a-4c93-b5be-7d0f4f0a0aae nodeName:}" failed. No retries permitted until 2026-09-09 15:20:44.519875903 +0000 UTC m=+10.118770530 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/26591625-285a-4c93-b5be-7d0f4f0a0aae-original-pull-secret") pod "global-pull-secret-syncer-kcl4l" (UID: "26591625-285a-4c93-b5be-7d0f4f0a0aae") : object "kube-system"/"original-pull-secret" not registered
Sep 09 15:20:40.620179 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:40.620137    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll8rs\" (UniqueName: \"kubernetes.io/projected/d830519d-a525-4029-ae78-d30bb04e19e5-kube-api-access-ll8rs\") pod \"network-check-target-xwx7m\" (UID: \"d830519d-a525-4029-ae78-d30bb04e19e5\") " pod="openshift-network-diagnostics/network-check-target-xwx7m"
Sep 09 15:20:40.620385 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:40.620352    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 15:20:40.620385 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:40.620374    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 15:20:40.620517 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:40.620388    2553 projected.go:196] Error preparing data for projected volume kube-api-access-ll8rs for pod openshift-network-diagnostics/network-check-target-xwx7m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:20:40.620517 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:40.620448    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d830519d-a525-4029-ae78-d30bb04e19e5-kube-api-access-ll8rs podName:d830519d-a525-4029-ae78-d30bb04e19e5 nodeName:}" failed. No retries permitted until 2026-09-09 15:20:44.620429044 +0000 UTC m=+10.219323672 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ll8rs" (UniqueName: "kubernetes.io/projected/d830519d-a525-4029-ae78-d30bb04e19e5-kube-api-access-ll8rs") pod "network-check-target-xwx7m" (UID: "d830519d-a525-4029-ae78-d30bb04e19e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:20:40.958618 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:40.958535    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5kbj"
Sep 09 15:20:40.959094 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:40.958689    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5kbj" podUID="57473bc0-f6d0-45f1-8954-93a7d2f4321b"
Sep 09 15:20:40.959094 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:40.958772    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcl4l"
Sep 09 15:20:40.959094 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:40.958885    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcl4l" podUID="26591625-285a-4c93-b5be-7d0f4f0a0aae"
Sep 09 15:20:41.956257 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:41.956006    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwx7m"
Sep 09 15:20:41.956257 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:41.956165    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xwx7m" podUID="d830519d-a525-4029-ae78-d30bb04e19e5"
Sep 09 15:20:42.959694 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:42.957617    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcl4l"
Sep 09 15:20:42.959694 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:42.957770    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcl4l" podUID="26591625-285a-4c93-b5be-7d0f4f0a0aae"
Sep 09 15:20:42.959694 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:42.958395    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5kbj"
Sep 09 15:20:42.959694 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:42.958518    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5kbj" podUID="57473bc0-f6d0-45f1-8954-93a7d2f4321b"
Sep 09 15:20:43.956331 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:43.955800    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwx7m"
Sep 09 15:20:43.956331 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:43.955952    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xwx7m" podUID="d830519d-a525-4029-ae78-d30bb04e19e5"
Sep 09 15:20:44.559540 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:44.559496    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57473bc0-f6d0-45f1-8954-93a7d2f4321b-metrics-certs\") pod \"network-metrics-daemon-v5kbj\" (UID: \"57473bc0-f6d0-45f1-8954-93a7d2f4321b\") " pod="openshift-multus/network-metrics-daemon-v5kbj"
Sep 09 15:20:44.560196 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:44.559603    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26591625-285a-4c93-b5be-7d0f4f0a0aae-original-pull-secret\") pod \"global-pull-secret-syncer-kcl4l\" (UID: \"26591625-285a-4c93-b5be-7d0f4f0a0aae\") " pod="kube-system/global-pull-secret-syncer-kcl4l"
Sep 09 15:20:44.560196 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:44.559726    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 15:20:44.560196 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:44.559805    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26591625-285a-4c93-b5be-7d0f4f0a0aae-original-pull-secret podName:26591625-285a-4c93-b5be-7d0f4f0a0aae nodeName:}" failed. No retries permitted until 2026-09-09 15:20:52.559788323 +0000 UTC m=+18.158682961 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/26591625-285a-4c93-b5be-7d0f4f0a0aae-original-pull-secret") pod "global-pull-secret-syncer-kcl4l" (UID: "26591625-285a-4c93-b5be-7d0f4f0a0aae") : object "kube-system"/"original-pull-secret" not registered
Sep 09 15:20:44.560196 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:44.559868    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:20:44.560196 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:44.559949    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57473bc0-f6d0-45f1-8954-93a7d2f4321b-metrics-certs podName:57473bc0-f6d0-45f1-8954-93a7d2f4321b nodeName:}" failed. No retries permitted until 2026-09-09 15:20:52.559927677 +0000 UTC m=+18.158822321 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57473bc0-f6d0-45f1-8954-93a7d2f4321b-metrics-certs") pod "network-metrics-daemon-v5kbj" (UID: "57473bc0-f6d0-45f1-8954-93a7d2f4321b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:20:44.660923 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:44.660786    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll8rs\" (UniqueName: \"kubernetes.io/projected/d830519d-a525-4029-ae78-d30bb04e19e5-kube-api-access-ll8rs\") pod \"network-check-target-xwx7m\" (UID: \"d830519d-a525-4029-ae78-d30bb04e19e5\") " pod="openshift-network-diagnostics/network-check-target-xwx7m"
Sep 09 15:20:44.661116 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:44.660974    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 15:20:44.661116 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:44.660998    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 15:20:44.661116 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:44.661011    2553 projected.go:196] Error preparing data for projected volume kube-api-access-ll8rs for pod openshift-network-diagnostics/network-check-target-xwx7m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:20:44.661116 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:44.661081    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d830519d-a525-4029-ae78-d30bb04e19e5-kube-api-access-ll8rs podName:d830519d-a525-4029-ae78-d30bb04e19e5 nodeName:}" failed. No retries permitted until 2026-09-09 15:20:52.661060282 +0000 UTC m=+18.259954919 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ll8rs" (UniqueName: "kubernetes.io/projected/d830519d-a525-4029-ae78-d30bb04e19e5-kube-api-access-ll8rs") pod "network-check-target-xwx7m" (UID: "d830519d-a525-4029-ae78-d30bb04e19e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:20:44.958252 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:44.957416    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcl4l"
Sep 09 15:20:44.958252 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:44.957558    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcl4l" podUID="26591625-285a-4c93-b5be-7d0f4f0a0aae"
Sep 09 15:20:44.959078 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:44.959015    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5kbj"
Sep 09 15:20:44.959204 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:44.959136    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5kbj" podUID="57473bc0-f6d0-45f1-8954-93a7d2f4321b"
Sep 09 15:20:45.956413 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:45.956378    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwx7m"
Sep 09 15:20:45.956902 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:45.956555    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xwx7m" podUID="d830519d-a525-4029-ae78-d30bb04e19e5"
Sep 09 15:20:46.956134 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:46.956091    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5kbj"
Sep 09 15:20:46.956319 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:46.956091    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcl4l"
Sep 09 15:20:46.956319 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:46.956242    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5kbj" podUID="57473bc0-f6d0-45f1-8954-93a7d2f4321b"
Sep 09 15:20:46.956669 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:46.956641    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcl4l" podUID="26591625-285a-4c93-b5be-7d0f4f0a0aae"
Sep 09 15:20:47.956569 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:47.956383    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwx7m"
Sep 09 15:20:47.956754 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:47.956660    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xwx7m" podUID="d830519d-a525-4029-ae78-d30bb04e19e5"
Sep 09 15:20:48.956742 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:48.956656    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcl4l"
Sep 09 15:20:48.957215 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:48.956663    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5kbj"
Sep 09 15:20:48.957215 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:48.956815    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcl4l" podUID="26591625-285a-4c93-b5be-7d0f4f0a0aae"
Sep 09 15:20:48.957215 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:48.956877    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5kbj" podUID="57473bc0-f6d0-45f1-8954-93a7d2f4321b"
Sep 09 15:20:49.955915 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:49.955882    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwx7m"
Sep 09 15:20:49.956078 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:49.955987    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xwx7m" podUID="d830519d-a525-4029-ae78-d30bb04e19e5"
Sep 09 15:20:50.955971 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:50.955934    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcl4l"
Sep 09 15:20:50.956407 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:50.956067    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcl4l" podUID="26591625-285a-4c93-b5be-7d0f4f0a0aae"
Sep 09 15:20:50.956407 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:50.956121    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5kbj"
Sep 09 15:20:50.956407 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:50.956220    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5kbj" podUID="57473bc0-f6d0-45f1-8954-93a7d2f4321b"
Sep 09 15:20:51.956206 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:51.956141    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwx7m"
Sep 09 15:20:51.956634 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:51.956281    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xwx7m" podUID="d830519d-a525-4029-ae78-d30bb04e19e5"
Sep 09 15:20:52.622254 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:52.622206    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26591625-285a-4c93-b5be-7d0f4f0a0aae-original-pull-secret\") pod \"global-pull-secret-syncer-kcl4l\" (UID: \"26591625-285a-4c93-b5be-7d0f4f0a0aae\") " pod="kube-system/global-pull-secret-syncer-kcl4l"
Sep 09 15:20:52.622442 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:52.622278    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57473bc0-f6d0-45f1-8954-93a7d2f4321b-metrics-certs\") pod \"network-metrics-daemon-v5kbj\" (UID: \"57473bc0-f6d0-45f1-8954-93a7d2f4321b\") " pod="openshift-multus/network-metrics-daemon-v5kbj"
Sep 09 15:20:52.622442 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:52.622426    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:20:52.622544 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:52.622488    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57473bc0-f6d0-45f1-8954-93a7d2f4321b-metrics-certs podName:57473bc0-f6d0-45f1-8954-93a7d2f4321b nodeName:}" failed. No retries permitted until 2026-09-09 15:21:08.622470068 +0000 UTC m=+34.221364696 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57473bc0-f6d0-45f1-8954-93a7d2f4321b-metrics-certs") pod "network-metrics-daemon-v5kbj" (UID: "57473bc0-f6d0-45f1-8954-93a7d2f4321b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:20:52.622708 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:52.622425    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 15:20:52.622806 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:52.622770    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26591625-285a-4c93-b5be-7d0f4f0a0aae-original-pull-secret podName:26591625-285a-4c93-b5be-7d0f4f0a0aae nodeName:}" failed. No retries permitted until 2026-09-09 15:21:08.622749604 +0000 UTC m=+34.221644234 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/26591625-285a-4c93-b5be-7d0f4f0a0aae-original-pull-secret") pod "global-pull-secret-syncer-kcl4l" (UID: "26591625-285a-4c93-b5be-7d0f4f0a0aae") : object "kube-system"/"original-pull-secret" not registered
Sep 09 15:20:52.723592 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:52.723552    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll8rs\" (UniqueName: \"kubernetes.io/projected/d830519d-a525-4029-ae78-d30bb04e19e5-kube-api-access-ll8rs\") pod \"network-check-target-xwx7m\" (UID: \"d830519d-a525-4029-ae78-d30bb04e19e5\") " pod="openshift-network-diagnostics/network-check-target-xwx7m"
Sep 09 15:20:52.723798 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:52.723688    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 15:20:52.723798 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:52.723711    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 15:20:52.723798 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:52.723724    2553 projected.go:196] Error preparing data for projected volume kube-api-access-ll8rs for pod openshift-network-diagnostics/network-check-target-xwx7m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:20:52.723959 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:52.723801    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d830519d-a525-4029-ae78-d30bb04e19e5-kube-api-access-ll8rs podName:d830519d-a525-4029-ae78-d30bb04e19e5 nodeName:}" failed. No retries permitted until 2026-09-09 15:21:08.723783587 +0000 UTC m=+34.322678209 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ll8rs" (UniqueName: "kubernetes.io/projected/d830519d-a525-4029-ae78-d30bb04e19e5-kube-api-access-ll8rs") pod "network-check-target-xwx7m" (UID: "d830519d-a525-4029-ae78-d30bb04e19e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:20:52.956413 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:52.956301    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5kbj"
Sep 09 15:20:52.956413 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:52.956320    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcl4l"
Sep 09 15:20:52.956903 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:52.956450    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5kbj" podUID="57473bc0-f6d0-45f1-8954-93a7d2f4321b"
Sep 09 15:20:52.956903 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:52.956520    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcl4l" podUID="26591625-285a-4c93-b5be-7d0f4f0a0aae"
Sep 09 15:20:53.956400 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:53.956365    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwx7m"
Sep 09 15:20:53.956580 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:53.956478    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xwx7m" podUID="d830519d-a525-4029-ae78-d30bb04e19e5"
Sep 09 15:20:54.957545 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:54.957481    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcl4l"
Sep 09 15:20:54.958029 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:54.957655    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcl4l" podUID="26591625-285a-4c93-b5be-7d0f4f0a0aae"
Sep 09 15:20:54.958029 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:54.957701    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5kbj"
Sep 09 15:20:54.958029 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:54.957825    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5kbj" podUID="57473bc0-f6d0-45f1-8954-93a7d2f4321b"
Sep 09 15:20:55.956364 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:55.956325    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwx7m"
Sep 09 15:20:55.956551 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:55.956445    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xwx7m" podUID="d830519d-a525-4029-ae78-d30bb04e19e5"
Sep 09 15:20:56.956716 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:56.956544    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcl4l"
Sep 09 15:20:56.957432 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:56.956616    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5kbj"
Sep 09 15:20:56.957432 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:56.956824    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcl4l" podUID="26591625-285a-4c93-b5be-7d0f4f0a0aae"
Sep 09 15:20:56.957432 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:56.956949    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5kbj" podUID="57473bc0-f6d0-45f1-8954-93a7d2f4321b"
Sep 09 15:20:57.035800 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:57.035767    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wgm9j" event={"ID":"3cfc8b11-7c62-46fc-8a5e-d5d889501cfd","Type":"ContainerStarted","Data":"109e0558962a20a5525c78e816580e7a35f590ba4a14cf4e38fdeadc1bee20e1"}
Sep 09 15:20:57.037283 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:57.037255    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bt4f2" event={"ID":"d1a7a818-24f1-4d7c-a3d5-416915c6c2cf","Type":"ContainerStarted","Data":"39a672c7550bb6cc007870132366b78f73600e8fdaafa069c3121349be01c6df"}
Sep 09 15:20:57.040670 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:57.040645    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-45pwd" event={"ID":"b60f4dd4-700c-4674-83ea-65b3d382f104","Type":"ContainerStarted","Data":"263ef32c21f88fd86853939720a1df7745e3a3ec5b0e14b6b6046b6dd0f222d4"}
Sep 09 15:20:57.040793 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:57.040757    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-45pwd" event={"ID":"b60f4dd4-700c-4674-83ea-65b3d382f104","Type":"ContainerStarted","Data":"354971a3d66af73f6fca2728238db166977585f5e2cbe206b7fee344345c9fc4"}
Sep 09 15:20:57.040863 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:57.040816    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-45pwd" event={"ID":"b60f4dd4-700c-4674-83ea-65b3d382f104","Type":"ContainerStarted","Data":"b376fb6f2279743af5b0b1e5f03ce4c130664290f38d7597e08eef8492888433"}
Sep 09 15:20:57.040917 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:57.040869    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-45pwd" event={"ID":"b60f4dd4-700c-4674-83ea-65b3d382f104","Type":"ContainerStarted","Data":"e96f1e6edb9932e87a9ecb4258f9d76076ce17ac0e7cd09a85808df53c7a5532"}
Sep 09 15:20:57.040966 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:57.040918    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-45pwd" event={"ID":"b60f4dd4-700c-4674-83ea-65b3d382f104","Type":"ContainerStarted","Data":"75b4401fcb1a58fd9878a0c93b775892970fdcfeaf5aa6d387bbd0e4ab2efcc9"}
Sep 09 15:20:57.042281 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:57.042258    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xjfqb" event={"ID":"3a444239-19ad-490a-a006-0a43ce90c9ef","Type":"ContainerStarted","Data":"6408409106d7bfaaae5f6104d3740c664ec7ae0f31925f13c63c90a1fa33174b"}
Sep 09 15:20:57.044483 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:57.044463    2553 generic.go:356] "Generic (PLEG): container finished" podID="4a532bf5-55b6-46a3-8c3f-4f3af4920a8d" containerID="f5890fb6d5e008c9a36c7085a5096fd48fe94998d0b350c669f3d0b08af67ef7" exitCode=0
Sep 09 15:20:57.044560 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:57.044526    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cmk65" event={"ID":"4a532bf5-55b6-46a3-8c3f-4f3af4920a8d","Type":"ContainerDied","Data":"f5890fb6d5e008c9a36c7085a5096fd48fe94998d0b350c669f3d0b08af67ef7"}
Sep 09 15:20:57.046421 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:57.046403    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhn2q" event={"ID":"d957458f-a014-49cb-9403-efba9f50e7f9","Type":"ContainerStarted","Data":"2dda88db1335251cef8b1e3fc8db3e622b6b7e1d6d3d054c19b2afedcae0c148"}
Sep 09 15:20:57.048142 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:57.048004    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-txxcj" event={"ID":"1aaf71f6-9195-4b50-a9a4-3cb85b10956a","Type":"ContainerStarted","Data":"a30986d2ce6d44079a2ae01a55642e3317c5559a8afc1a5d288b025c8f6343de"}
Sep 09 15:20:57.050046 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:57.050017    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fwsrs" event={"ID":"587d946b-e46a-4e99-84bf-8ef93be089c9","Type":"ContainerStarted","Data":"546f598d87a44a4649f72b274cbd6448a4e72dec9347d3476281a950e5cfab9d"}
Sep 09 15:20:57.050458 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:57.050411    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-170.ec2.internal" podStartSLOduration=21.050399483 podStartE2EDuration="21.050399483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:20:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:20:39.013486311 +0000 UTC m=+4.612380959" watchObservedRunningTime="2026-09-09 15:20:57.050399483 +0000 UTC m=+22.649294127"
Sep 09 15:20:57.050785 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:57.050711    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wgm9j" podStartSLOduration=2.854227806 podStartE2EDuration="22.050699159s" totalImagesPullingTime="19.196471353s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:20:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:20:57.049877531 +0000 UTC m=+22.648772175" watchObservedRunningTime="2026-09-09 15:20:57.050699159 +0000 UTC m=+22.649593804"
Sep 09 15:20:57.051819 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:57.051792    2553 generic.go:356] "Generic (PLEG): container finished" podID="17b2b5583df839506650e89cca8d5eb6" containerID="b9fd5abaa9b156195654c952c6d957569ddf9411f5571214c2becdf0238cc609" exitCode=0
Sep 09 15:20:57.051911 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:57.051827    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-170.ec2.internal" event={"ID":"17b2b5583df839506650e89cca8d5eb6","Type":"ContainerDied","Data":"b9fd5abaa9b156195654c952c6d957569ddf9411f5571214c2becdf0238cc609"}
Sep 09 15:20:57.065630 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:57.065591    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-bt4f2" podStartSLOduration=2.740827529 podStartE2EDuration="22.065576921s" totalImagesPullingTime="19.324749392s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:20:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:20:57.065442856 +0000 UTC m=+22.664337500" watchObservedRunningTime="2026-09-09 15:20:57.065576921 +0000 UTC m=+22.664471564"
Sep 09 15:20:57.078791 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:57.078751    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-txxcj" podStartSLOduration=2.835970568 podStartE2EDuration="22.078723854s" totalImagesPullingTime="19.242753286s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:20:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:20:57.078444276 +0000 UTC m=+22.677338910" watchObservedRunningTime="2026-09-09 15:20:57.078723854 +0000 UTC m=+22.677618497"
Sep 09 15:20:57.091114 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:57.091073    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xjfqb" podStartSLOduration=2.856516583 podStartE2EDuration="22.091053273s" totalImagesPullingTime="19.23453669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:20:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:20:57.090997414 +0000 UTC m=+22.689892058" watchObservedRunningTime="2026-09-09 15:20:57.091053273 +0000 UTC m=+22.689947916"
Sep 09 15:20:57.133057 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:57.132996    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fwsrs" podStartSLOduration=2.809836813 podStartE2EDuration="22.132971577s" totalImagesPullingTime="19.323134764s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:20:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:20:57.132495276 +0000 UTC m=+22.731389921" watchObservedRunningTime="2026-09-09 15:20:57.132971577 +0000 UTC m=+22.731866223"
Sep 09 15:20:57.956607 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:57.956569    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwx7m"
Sep 09 15:20:57.956912 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:57.956704    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xwx7m" podUID="d830519d-a525-4029-ae78-d30bb04e19e5"
Sep 09 15:20:58.056235 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:58.056037    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-thqt6" event={"ID":"436fdcf3-5cd3-44e0-b5b8-3b878c58f6c7","Type":"ContainerStarted","Data":"c7e77f59ddbe93509aeba65c872fd0e992e1e8324991a413485d622e95b16b47"}
Sep 09 15:20:58.060330 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:58.060300    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-45pwd" event={"ID":"b60f4dd4-700c-4674-83ea-65b3d382f104","Type":"ContainerStarted","Data":"7954cafd839aa16670e96315f82d04f67048b74f9ea873ee48067f1e68ce1ad5"}
Sep 09 15:20:58.071044 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:58.071003    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-thqt6" podStartSLOduration=3.818028317 podStartE2EDuration="23.07098752s" totalImagesPullingTime="19.252959203s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:20:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:20:58.070582354 +0000 UTC m=+23.669476999" watchObservedRunningTime="2026-09-09 15:20:58.07098752 +0000 UTC m=+23.669882165"
Sep 09 15:20:58.180289 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:58.180169    2553 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 15:20:58.738830 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:58.738792    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xjfqb"
Sep 09 15:20:58.739751 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:58.739707    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xjfqb"
Sep 09 15:20:58.837411 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:58.837305    2553 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T15:20:58.180185719Z","UUID":"0514036d-6c95-40d6-9356-68b09d97b659","Handler":null,"Name":"","Endpoint":""}
Sep 09 15:20:58.839210 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:58.839176    2553 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 15:20:58.839210 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:58.839212    2553 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 15:20:58.956380 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:58.956349    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcl4l"
Sep 09 15:20:58.956380 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:58.956393    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5kbj"
Sep 09 15:20:58.956609 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:58.956486    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcl4l" podUID="26591625-285a-4c93-b5be-7d0f4f0a0aae"
Sep 09 15:20:58.956670 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:58.956641    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5kbj" podUID="57473bc0-f6d0-45f1-8954-93a7d2f4321b"
Sep 09 15:20:59.063777 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:59.063685    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhn2q" event={"ID":"d957458f-a014-49cb-9403-efba9f50e7f9","Type":"ContainerStarted","Data":"d55399462a59be5c547a5f1d1633d7ab10868c1f2c65746a98d89d7199bdb10f"}
Sep 09 15:20:59.065257 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:59.065233    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-170.ec2.internal" event={"ID":"17b2b5583df839506650e89cca8d5eb6","Type":"ContainerStarted","Data":"e80925740421d95264e6659b4b9843655f02f9cfec2b382fe767cbdf52a6d143"}
Sep 09 15:20:59.065636 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:59.065601    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xjfqb"
Sep 09 15:20:59.065951 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:59.065933    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xjfqb"
Sep 09 15:20:59.079914 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:59.079862    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-170.ec2.internal" podStartSLOduration=23.079843555 podStartE2EDuration="23.079843555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:20:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:20:59.079288682 +0000 UTC m=+24.678183329" watchObservedRunningTime="2026-09-09 15:20:59.079843555 +0000 UTC m=+24.678738201"
Sep 09 15:20:59.956536 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:20:59.956506    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwx7m"
Sep 09 15:20:59.956759 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:20:59.956594    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xwx7m" podUID="d830519d-a525-4029-ae78-d30bb04e19e5"
Sep 09 15:21:00.070090 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:00.070032    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhn2q" event={"ID":"d957458f-a014-49cb-9403-efba9f50e7f9","Type":"ContainerStarted","Data":"614f30182fd2906743db3f7476bc50f490ba260ca4833073e49d9655b9bec46f"}
Sep 09 15:21:00.073956 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:00.073925    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-45pwd" event={"ID":"b60f4dd4-700c-4674-83ea-65b3d382f104","Type":"ContainerStarted","Data":"fc5a936a3a98fa5660109a6caed543148a0117c8296c7feb662d61a7cc2b079e"}
Sep 09 15:21:00.956852 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:00.956808    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcl4l"
Sep 09 15:21:00.957039 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:00.956855    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5kbj"
Sep 09 15:21:00.957039 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:21:00.956946    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcl4l" podUID="26591625-285a-4c93-b5be-7d0f4f0a0aae"
Sep 09 15:21:00.957164 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:21:00.957074    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5kbj" podUID="57473bc0-f6d0-45f1-8954-93a7d2f4321b"
Sep 09 15:21:01.956116 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:01.955904    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwx7m"
Sep 09 15:21:01.956465 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:21:01.956157    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xwx7m" podUID="d830519d-a525-4029-ae78-d30bb04e19e5"
Sep 09 15:21:02.081956 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:02.081916    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-45pwd" event={"ID":"b60f4dd4-700c-4674-83ea-65b3d382f104","Type":"ContainerStarted","Data":"3cf1ae95d5be060b5fcb2f041ca4f423706646cf81a4bcdc9e2647938ef1399e"}
Sep 09 15:21:02.082281 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:02.082236    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:21:02.082382 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:02.082313    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:21:02.084021 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:02.083992    2553 generic.go:356] "Generic (PLEG): container finished" podID="4a532bf5-55b6-46a3-8c3f-4f3af4920a8d" containerID="d142f77fbce2a2eddd715293826ae849fcda43182d6cd838219c09d216bdd434" exitCode=0
Sep 09 15:21:02.084132 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:02.084046    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cmk65" event={"ID":"4a532bf5-55b6-46a3-8c3f-4f3af4920a8d","Type":"ContainerDied","Data":"d142f77fbce2a2eddd715293826ae849fcda43182d6cd838219c09d216bdd434"}
Sep 09 15:21:02.097712 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:02.097691    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:21:02.097848 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:02.097840    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:21:02.109182 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:02.109141    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-45pwd" podStartSLOduration=7.78719007 podStartE2EDuration="27.109126934s" totalImagesPullingTime="19.321936864s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:20:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:21:02.107796709 +0000 UTC m=+27.706691353" watchObservedRunningTime="2026-09-09 15:21:02.109126934 +0000 UTC m=+27.708021579"
Sep 09 15:21:02.109370 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:02.109352    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhn2q" podStartSLOduration=5.374687356 podStartE2EDuration="27.109346019s" totalImagesPullingTime="21.734658663s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:20:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:21:00.087603402 +0000 UTC m=+25.686498065" watchObservedRunningTime="2026-09-09 15:21:02.109346019 +0000 UTC m=+27.708240700"
Sep 09 15:21:02.958831 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:02.958795    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcl4l"
Sep 09 15:21:02.959500 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:21:02.958917    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcl4l" podUID="26591625-285a-4c93-b5be-7d0f4f0a0aae"
Sep 09 15:21:02.959500 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:02.959339    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5kbj"
Sep 09 15:21:02.959500 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:21:02.959437    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5kbj" podUID="57473bc0-f6d0-45f1-8954-93a7d2f4321b"
Sep 09 15:21:03.086315 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:03.086283    2553 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 09 15:21:03.618073 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:03.618033    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v5kbj"]
Sep 09 15:21:03.618208 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:03.618162    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5kbj"
Sep 09 15:21:03.618296 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:21:03.618270    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5kbj" podUID="57473bc0-f6d0-45f1-8954-93a7d2f4321b"
Sep 09 15:21:03.618629 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:03.618610    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kcl4l"]
Sep 09 15:21:03.618694 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:03.618684    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcl4l"
Sep 09 15:21:03.618794 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:21:03.618775    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcl4l" podUID="26591625-285a-4c93-b5be-7d0f4f0a0aae"
Sep 09 15:21:03.628130 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:03.628094    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xwx7m"]
Sep 09 15:21:03.628256 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:03.628191    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwx7m"
Sep 09 15:21:03.628304 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:21:03.628268    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xwx7m" podUID="d830519d-a525-4029-ae78-d30bb04e19e5"
Sep 09 15:21:04.088386 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:04.088356    2553 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 09 15:21:04.956915 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:04.956724    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwx7m"
Sep 09 15:21:04.957053 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:21:04.956976    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xwx7m" podUID="d830519d-a525-4029-ae78-d30bb04e19e5"
Sep 09 15:21:05.092555 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:05.092515    2553 generic.go:356] "Generic (PLEG): container finished" podID="4a532bf5-55b6-46a3-8c3f-4f3af4920a8d" containerID="ec621b459a52f728a212eb66c8dd5dfbdf0e7b705e0c46027bc302715f778fb2" exitCode=0
Sep 09 15:21:05.093025 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:05.092599    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cmk65" event={"ID":"4a532bf5-55b6-46a3-8c3f-4f3af4920a8d","Type":"ContainerDied","Data":"ec621b459a52f728a212eb66c8dd5dfbdf0e7b705e0c46027bc302715f778fb2"}
Sep 09 15:21:05.956251 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:05.956218    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5kbj"
Sep 09 15:21:05.956251 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:05.956251    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcl4l"
Sep 09 15:21:05.956430 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:21:05.956322    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5kbj" podUID="57473bc0-f6d0-45f1-8954-93a7d2f4321b"
Sep 09 15:21:05.956472 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:21:05.956423    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcl4l" podUID="26591625-285a-4c93-b5be-7d0f4f0a0aae"
Sep 09 15:21:06.097419 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:06.097383    2553 generic.go:356] "Generic (PLEG): container finished" podID="4a532bf5-55b6-46a3-8c3f-4f3af4920a8d" containerID="58f072fb8cf39a91e03f44f0a0c2799868f9262bae156891fd179a9f60d55789" exitCode=0
Sep 09 15:21:06.097796 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:06.097439    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cmk65" event={"ID":"4a532bf5-55b6-46a3-8c3f-4f3af4920a8d","Type":"ContainerDied","Data":"58f072fb8cf39a91e03f44f0a0c2799868f9262bae156891fd179a9f60d55789"}
Sep 09 15:21:06.410169 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:06.410075    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:21:06.410296 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:06.410276    2553 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 09 15:21:06.419933 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:06.419897    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-45pwd" podUID="b60f4dd4-700c-4674-83ea-65b3d382f104" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 09 15:21:06.428336 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:06.428307    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-45pwd" podUID="b60f4dd4-700c-4674-83ea-65b3d382f104" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 09 15:21:06.956819 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:06.956782    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwx7m"
Sep 09 15:21:06.956985 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:21:06.956896    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xwx7m" podUID="d830519d-a525-4029-ae78-d30bb04e19e5"
Sep 09 15:21:07.956644 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:07.956611    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcl4l"
Sep 09 15:21:07.957055 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:07.956611    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5kbj"
Sep 09 15:21:07.957055 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:21:07.956780    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcl4l" podUID="26591625-285a-4c93-b5be-7d0f4f0a0aae"
Sep 09 15:21:07.957055 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:21:07.956820    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5kbj" podUID="57473bc0-f6d0-45f1-8954-93a7d2f4321b"
Sep 09 15:21:08.638996 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:08.638956    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26591625-285a-4c93-b5be-7d0f4f0a0aae-original-pull-secret\") pod \"global-pull-secret-syncer-kcl4l\" (UID: \"26591625-285a-4c93-b5be-7d0f4f0a0aae\") " pod="kube-system/global-pull-secret-syncer-kcl4l"
Sep 09 15:21:08.639251 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:08.639024    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57473bc0-f6d0-45f1-8954-93a7d2f4321b-metrics-certs\") pod \"network-metrics-daemon-v5kbj\" (UID: \"57473bc0-f6d0-45f1-8954-93a7d2f4321b\") " pod="openshift-multus/network-metrics-daemon-v5kbj"
Sep 09 15:21:08.639251 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:21:08.639120    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 15:21:08.639251 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:21:08.639183    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26591625-285a-4c93-b5be-7d0f4f0a0aae-original-pull-secret podName:26591625-285a-4c93-b5be-7d0f4f0a0aae nodeName:}" failed. No retries permitted until 2026-09-09 15:21:40.639168868 +0000 UTC m=+66.238063494 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/26591625-285a-4c93-b5be-7d0f4f0a0aae-original-pull-secret") pod "global-pull-secret-syncer-kcl4l" (UID: "26591625-285a-4c93-b5be-7d0f4f0a0aae") : object "kube-system"/"original-pull-secret" not registered
Sep 09 15:21:08.639251 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:21:08.639128    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:21:08.639251 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:21:08.639225    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57473bc0-f6d0-45f1-8954-93a7d2f4321b-metrics-certs podName:57473bc0-f6d0-45f1-8954-93a7d2f4321b nodeName:}" failed. No retries permitted until 2026-09-09 15:21:40.639216496 +0000 UTC m=+66.238111119 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57473bc0-f6d0-45f1-8954-93a7d2f4321b-metrics-certs") pod "network-metrics-daemon-v5kbj" (UID: "57473bc0-f6d0-45f1-8954-93a7d2f4321b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:21:08.740030 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:08.739997    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll8rs\" (UniqueName: \"kubernetes.io/projected/d830519d-a525-4029-ae78-d30bb04e19e5-kube-api-access-ll8rs\") pod \"network-check-target-xwx7m\" (UID: \"d830519d-a525-4029-ae78-d30bb04e19e5\") " pod="openshift-network-diagnostics/network-check-target-xwx7m"
Sep 09 15:21:08.740208 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:21:08.740164    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 15:21:08.740208 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:21:08.740185    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 15:21:08.740208 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:21:08.740196    2553 projected.go:196] Error preparing data for projected volume kube-api-access-ll8rs for pod openshift-network-diagnostics/network-check-target-xwx7m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:21:08.740374 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:21:08.740253    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d830519d-a525-4029-ae78-d30bb04e19e5-kube-api-access-ll8rs podName:d830519d-a525-4029-ae78-d30bb04e19e5 nodeName:}" failed. No retries permitted until 2026-09-09 15:21:40.740234916 +0000 UTC m=+66.339129542 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ll8rs" (UniqueName: "kubernetes.io/projected/d830519d-a525-4029-ae78-d30bb04e19e5-kube-api-access-ll8rs") pod "network-check-target-xwx7m" (UID: "d830519d-a525-4029-ae78-d30bb04e19e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:21:08.956492 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:08.956461    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwx7m"
Sep 09 15:21:08.956623 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:21:08.956576    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xwx7m" podUID="d830519d-a525-4029-ae78-d30bb04e19e5"
Sep 09 15:21:09.094685 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.094595    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-170.ec2.internal" event="NodeReady"
Sep 09 15:21:09.095069 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.094815    2553 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 15:21:09.137423 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.137385    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-cwkld"]
Sep 09 15:21:09.152043 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.151806    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7kz9s"]
Sep 09 15:21:09.152043 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.151993    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cwkld"
Sep 09 15:21:09.154543 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.154490    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 15:21:09.154543 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.154509    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jnqpd\""
Sep 09 15:21:09.154753 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.154490    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 15:21:09.165861 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.165838    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cwkld"]
Sep 09 15:21:09.166023 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.165883    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7kz9s"]
Sep 09 15:21:09.166023 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.165960    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7kz9s"
Sep 09 15:21:09.168568 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.168376    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 15:21:09.168568 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.168449    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 15:21:09.168568 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.168382    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mczr8\""
Sep 09 15:21:09.168869 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.168601    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 15:21:09.245911 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.245824    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/acc8de36-2e42-4471-a31e-38c66f87ab8c-config-volume\") pod \"dns-default-cwkld\" (UID: \"acc8de36-2e42-4471-a31e-38c66f87ab8c\") " pod="openshift-dns/dns-default-cwkld"
Sep 09 15:21:09.245911 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.245891    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acc8de36-2e42-4471-a31e-38c66f87ab8c-tmp-dir\") pod \"dns-default-cwkld\" (UID: \"acc8de36-2e42-4471-a31e-38c66f87ab8c\") " pod="openshift-dns/dns-default-cwkld"
Sep 09 15:21:09.246114 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.246009    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acc8de36-2e42-4471-a31e-38c66f87ab8c-metrics-tls\") pod \"dns-default-cwkld\" (UID: \"acc8de36-2e42-4471-a31e-38c66f87ab8c\") " pod="openshift-dns/dns-default-cwkld"
Sep 09 15:21:09.246114 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.246058    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtgwb\" (UniqueName: \"kubernetes.io/projected/acc8de36-2e42-4471-a31e-38c66f87ab8c-kube-api-access-dtgwb\") pod \"dns-default-cwkld\" (UID: \"acc8de36-2e42-4471-a31e-38c66f87ab8c\") " pod="openshift-dns/dns-default-cwkld"
Sep 09 15:21:09.347392 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.347350    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acc8de36-2e42-4471-a31e-38c66f87ab8c-tmp-dir\") pod \"dns-default-cwkld\" (UID: \"acc8de36-2e42-4471-a31e-38c66f87ab8c\") " pod="openshift-dns/dns-default-cwkld"
Sep 09 15:21:09.347560 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.347423    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acc8de36-2e42-4471-a31e-38c66f87ab8c-metrics-tls\") pod \"dns-default-cwkld\" (UID: \"acc8de36-2e42-4471-a31e-38c66f87ab8c\") " pod="openshift-dns/dns-default-cwkld"
Sep 09 15:21:09.347560 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.347464    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2082125-0643-4b68-a968-a45d7d8aac36-cert\") pod \"ingress-canary-7kz9s\" (UID: \"d2082125-0643-4b68-a968-a45d7d8aac36\") " pod="openshift-ingress-canary/ingress-canary-7kz9s"
Sep 09 15:21:09.347560 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.347498    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtgwb\" (UniqueName: \"kubernetes.io/projected/acc8de36-2e42-4471-a31e-38c66f87ab8c-kube-api-access-dtgwb\") pod \"dns-default-cwkld\" (UID: \"acc8de36-2e42-4471-a31e-38c66f87ab8c\") " pod="openshift-dns/dns-default-cwkld"
Sep 09 15:21:09.347560 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.347527    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/acc8de36-2e42-4471-a31e-38c66f87ab8c-config-volume\") pod \"dns-default-cwkld\" (UID: \"acc8de36-2e42-4471-a31e-38c66f87ab8c\") " pod="openshift-dns/dns-default-cwkld"
Sep 09 15:21:09.347560 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.347549    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcnt6\" (UniqueName: \"kubernetes.io/projected/d2082125-0643-4b68-a968-a45d7d8aac36-kube-api-access-kcnt6\") pod \"ingress-canary-7kz9s\" (UID: \"d2082125-0643-4b68-a968-a45d7d8aac36\") " pod="openshift-ingress-canary/ingress-canary-7kz9s"
Sep 09 15:21:09.347900 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:21:09.347609    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 15:21:09.347900 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:21:09.347687    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acc8de36-2e42-4471-a31e-38c66f87ab8c-metrics-tls podName:acc8de36-2e42-4471-a31e-38c66f87ab8c nodeName:}" failed. No retries permitted until 2026-09-09 15:21:09.847664918 +0000 UTC m=+35.446559579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acc8de36-2e42-4471-a31e-38c66f87ab8c-metrics-tls") pod "dns-default-cwkld" (UID: "acc8de36-2e42-4471-a31e-38c66f87ab8c") : secret "dns-default-metrics-tls" not found
Sep 09 15:21:09.347900 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.347757    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acc8de36-2e42-4471-a31e-38c66f87ab8c-tmp-dir\") pod \"dns-default-cwkld\" (UID: \"acc8de36-2e42-4471-a31e-38c66f87ab8c\") " pod="openshift-dns/dns-default-cwkld"
Sep 09 15:21:09.348047 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.348029    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/acc8de36-2e42-4471-a31e-38c66f87ab8c-config-volume\") pod \"dns-default-cwkld\" (UID: \"acc8de36-2e42-4471-a31e-38c66f87ab8c\") " pod="openshift-dns/dns-default-cwkld"
Sep 09 15:21:09.358916 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.358885    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtgwb\" (UniqueName: \"kubernetes.io/projected/acc8de36-2e42-4471-a31e-38c66f87ab8c-kube-api-access-dtgwb\") pod \"dns-default-cwkld\" (UID: \"acc8de36-2e42-4471-a31e-38c66f87ab8c\") " pod="openshift-dns/dns-default-cwkld"
Sep 09 15:21:09.448952 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.448907    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcnt6\" (UniqueName: \"kubernetes.io/projected/d2082125-0643-4b68-a968-a45d7d8aac36-kube-api-access-kcnt6\") pod \"ingress-canary-7kz9s\" (UID: \"d2082125-0643-4b68-a968-a45d7d8aac36\") " pod="openshift-ingress-canary/ingress-canary-7kz9s"
Sep 09 15:21:09.449145 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.449055    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2082125-0643-4b68-a968-a45d7d8aac36-cert\") pod \"ingress-canary-7kz9s\" (UID: \"d2082125-0643-4b68-a968-a45d7d8aac36\") " pod="openshift-ingress-canary/ingress-canary-7kz9s"
Sep 09 15:21:09.452254 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.452223    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2082125-0643-4b68-a968-a45d7d8aac36-cert\") pod \"ingress-canary-7kz9s\" (UID: \"d2082125-0643-4b68-a968-a45d7d8aac36\") " pod="openshift-ingress-canary/ingress-canary-7kz9s"
Sep 09 15:21:09.458221 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.458192    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcnt6\" (UniqueName: \"kubernetes.io/projected/d2082125-0643-4b68-a968-a45d7d8aac36-kube-api-access-kcnt6\") pod \"ingress-canary-7kz9s\" (UID: \"d2082125-0643-4b68-a968-a45d7d8aac36\") " pod="openshift-ingress-canary/ingress-canary-7kz9s"
Sep 09 15:21:09.474993 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.474965    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7kz9s"
Sep 09 15:21:09.790274 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.790198    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7kz9s"]
Sep 09 15:21:09.795409 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:21:09.795375    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2082125_0643_4b68_a968_a45d7d8aac36.slice/crio-059b1e021281084e834ff057f14b151d92d556d0ee712144b891ae91150bcb11 WatchSource:0}: Error finding container 059b1e021281084e834ff057f14b151d92d556d0ee712144b891ae91150bcb11: Status 404 returned error can't find the container with id 059b1e021281084e834ff057f14b151d92d556d0ee712144b891ae91150bcb11
Sep 09 15:21:09.851435 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.851398    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acc8de36-2e42-4471-a31e-38c66f87ab8c-metrics-tls\") pod \"dns-default-cwkld\" (UID: \"acc8de36-2e42-4471-a31e-38c66f87ab8c\") " pod="openshift-dns/dns-default-cwkld"
Sep 09 15:21:09.851604 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:21:09.851554    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 15:21:09.851666 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:21:09.851632    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acc8de36-2e42-4471-a31e-38c66f87ab8c-metrics-tls podName:acc8de36-2e42-4471-a31e-38c66f87ab8c nodeName:}" failed. No retries permitted until 2026-09-09 15:21:10.851606864 +0000 UTC m=+36.450501496 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acc8de36-2e42-4471-a31e-38c66f87ab8c-metrics-tls") pod "dns-default-cwkld" (UID: "acc8de36-2e42-4471-a31e-38c66f87ab8c") : secret "dns-default-metrics-tls" not found
Sep 09 15:21:09.956843 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.956807    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcl4l"
Sep 09 15:21:09.957025 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.956821    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5kbj"
Sep 09 15:21:09.959645 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.959405    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 15:21:09.959645 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.959429    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xdr6q\""
Sep 09 15:21:09.959645 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:09.959470    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 15:21:10.107872 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:10.107785    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7kz9s" event={"ID":"d2082125-0643-4b68-a968-a45d7d8aac36","Type":"ContainerStarted","Data":"059b1e021281084e834ff057f14b151d92d556d0ee712144b891ae91150bcb11"}
Sep 09 15:21:10.859656 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:10.859621    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acc8de36-2e42-4471-a31e-38c66f87ab8c-metrics-tls\") pod \"dns-default-cwkld\" (UID: \"acc8de36-2e42-4471-a31e-38c66f87ab8c\") " pod="openshift-dns/dns-default-cwkld"
Sep 09 15:21:10.859854 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:21:10.859759    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 15:21:10.859854 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:21:10.859820    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acc8de36-2e42-4471-a31e-38c66f87ab8c-metrics-tls podName:acc8de36-2e42-4471-a31e-38c66f87ab8c nodeName:}" failed. No retries permitted until 2026-09-09 15:21:12.859801124 +0000 UTC m=+38.458695746 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acc8de36-2e42-4471-a31e-38c66f87ab8c-metrics-tls") pod "dns-default-cwkld" (UID: "acc8de36-2e42-4471-a31e-38c66f87ab8c") : secret "dns-default-metrics-tls" not found
Sep 09 15:21:10.956259 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:10.956224    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwx7m"
Sep 09 15:21:10.960759 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:10.959305    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 15:21:10.960759 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:10.959343    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7889l\""
Sep 09 15:21:10.960759 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:10.960310    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 15:21:12.875581 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:12.875542    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acc8de36-2e42-4471-a31e-38c66f87ab8c-metrics-tls\") pod \"dns-default-cwkld\" (UID: \"acc8de36-2e42-4471-a31e-38c66f87ab8c\") " pod="openshift-dns/dns-default-cwkld"
Sep 09 15:21:12.876096 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:21:12.875750    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 15:21:12.876096 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:21:12.875840    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acc8de36-2e42-4471-a31e-38c66f87ab8c-metrics-tls podName:acc8de36-2e42-4471-a31e-38c66f87ab8c nodeName:}" failed. No retries permitted until 2026-09-09 15:21:16.875816678 +0000 UTC m=+42.474711304 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acc8de36-2e42-4471-a31e-38c66f87ab8c-metrics-tls") pod "dns-default-cwkld" (UID: "acc8de36-2e42-4471-a31e-38c66f87ab8c") : secret "dns-default-metrics-tls" not found
Sep 09 15:21:14.120561 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:14.120476    2553 generic.go:356] "Generic (PLEG): container finished" podID="4a532bf5-55b6-46a3-8c3f-4f3af4920a8d" containerID="c1b7a93804f6cc3b89f207fe9ff1897b35c4225b464ba804597e56d4e657a880" exitCode=0
Sep 09 15:21:14.121221 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:14.120553    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cmk65" event={"ID":"4a532bf5-55b6-46a3-8c3f-4f3af4920a8d","Type":"ContainerDied","Data":"c1b7a93804f6cc3b89f207fe9ff1897b35c4225b464ba804597e56d4e657a880"}
Sep 09 15:21:14.122186 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:14.122165    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7kz9s" event={"ID":"d2082125-0643-4b68-a968-a45d7d8aac36","Type":"ContainerStarted","Data":"51d300330a7a4565767a9343b686f1ac2f302369843d5d8a9e856d39c806654a"}
Sep 09 15:21:14.153597 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:14.153554    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7kz9s" podStartSLOduration=1.319430867 podStartE2EDuration="5.153539519s" totalImagesPullingTime="3.834108652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:21:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:21:14.152876405 +0000 UTC m=+39.751771051" watchObservedRunningTime="2026-09-09 15:21:14.153539519 +0000 UTC m=+39.752434160"
Sep 09 15:21:15.128161 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:15.127973    2553 generic.go:356] "Generic (PLEG): container finished" podID="4a532bf5-55b6-46a3-8c3f-4f3af4920a8d" containerID="f9aa50d9960374cf6199e4e3c2b1b26cc554a587947af886908968ac01f523ab" exitCode=0
Sep 09 15:21:15.128161 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:15.128058    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cmk65" event={"ID":"4a532bf5-55b6-46a3-8c3f-4f3af4920a8d","Type":"ContainerDied","Data":"f9aa50d9960374cf6199e4e3c2b1b26cc554a587947af886908968ac01f523ab"}
Sep 09 15:21:16.133775 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:16.133720    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cmk65" event={"ID":"4a532bf5-55b6-46a3-8c3f-4f3af4920a8d","Type":"ContainerStarted","Data":"9d4b3841330178b83859c67c7d01d25dfaa53ce128f096affd8e7a9ab2bb51d3"}
Sep 09 15:21:16.154425 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:16.154378    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cmk65" podStartSLOduration=6.416203037 podStartE2EDuration="41.154361764s" totalImagesPullingTime="34.738158727s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:20:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:21:16.152918947 +0000 UTC m=+41.751813602" watchObservedRunningTime="2026-09-09 15:21:16.154361764 +0000 UTC m=+41.753256407"
Sep 09 15:21:16.906939 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:16.906899    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acc8de36-2e42-4471-a31e-38c66f87ab8c-metrics-tls\") pod \"dns-default-cwkld\" (UID: \"acc8de36-2e42-4471-a31e-38c66f87ab8c\") " pod="openshift-dns/dns-default-cwkld"
Sep 09 15:21:16.907125 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:21:16.907021    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 15:21:16.907125 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:21:16.907080    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acc8de36-2e42-4471-a31e-38c66f87ab8c-metrics-tls podName:acc8de36-2e42-4471-a31e-38c66f87ab8c nodeName:}" failed. No retries permitted until 2026-09-09 15:21:24.907060989 +0000 UTC m=+50.505955614 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acc8de36-2e42-4471-a31e-38c66f87ab8c-metrics-tls") pod "dns-default-cwkld" (UID: "acc8de36-2e42-4471-a31e-38c66f87ab8c") : secret "dns-default-metrics-tls" not found
Sep 09 15:21:24.962832 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:24.962799    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acc8de36-2e42-4471-a31e-38c66f87ab8c-metrics-tls\") pod \"dns-default-cwkld\" (UID: \"acc8de36-2e42-4471-a31e-38c66f87ab8c\") " pod="openshift-dns/dns-default-cwkld"
Sep 09 15:21:24.963313 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:21:24.962915    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 15:21:24.963313 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:21:24.962968    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acc8de36-2e42-4471-a31e-38c66f87ab8c-metrics-tls podName:acc8de36-2e42-4471-a31e-38c66f87ab8c nodeName:}" failed. No retries permitted until 2026-09-09 15:21:40.962956019 +0000 UTC m=+66.561850641 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acc8de36-2e42-4471-a31e-38c66f87ab8c-metrics-tls") pod "dns-default-cwkld" (UID: "acc8de36-2e42-4471-a31e-38c66f87ab8c") : secret "dns-default-metrics-tls" not found
Sep 09 15:21:30.892009 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:30.891973    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77676775cb-ckrdk"]
Sep 09 15:21:30.893886 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:30.893869    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77676775cb-ckrdk"
Sep 09 15:21:30.897855 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:30.897833    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 09 15:21:30.897979 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:30.897871    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 15:21:30.897979 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:30.897898    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 15:21:30.897979 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:30.897956    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 15:21:30.898116 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:30.897986    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-wt7ns\""
Sep 09 15:21:30.905078 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:30.905053    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77676775cb-ckrdk"]
Sep 09 15:21:30.907983 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:30.907954    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6967d4f-t7dwt"]
Sep 09 15:21:30.909679 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:30.909662    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6967d4f-t7dwt"
Sep 09 15:21:30.912314 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:30.912295    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 09 15:21:30.912314 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:30.912305    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 09 15:21:30.912314 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:30.912312    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 09 15:21:30.912496 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:30.912315    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 09 15:21:30.923111 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:30.923087    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6967d4f-t7dwt"]
Sep 09 15:21:31.005192 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:31.005154    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ed49a29-af63-4d51-8729-eb4fcc081daa-additional-service-ca\") pod \"cluster-proxy-proxy-agent-86d6967d4f-t7dwt\" (UID: \"8ed49a29-af63-4d51-8729-eb4fcc081daa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6967d4f-t7dwt"
Sep 09 15:21:31.005192 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:31.005208    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/61925eed-e9ab-4486-b178-9e826d014af4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-77676775cb-ckrdk\" (UID: \"61925eed-e9ab-4486-b178-9e826d014af4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77676775cb-ckrdk"
Sep 09 15:21:31.005488 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:31.005228    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x86j\" (UniqueName: \"kubernetes.io/projected/8ed49a29-af63-4d51-8729-eb4fcc081daa-kube-api-access-9x86j\") pod \"cluster-proxy-proxy-agent-86d6967d4f-t7dwt\" (UID: \"8ed49a29-af63-4d51-8729-eb4fcc081daa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6967d4f-t7dwt"
Sep 09 15:21:31.005488 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:31.005251    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8ed49a29-af63-4d51-8729-eb4fcc081daa-ca\") pod \"cluster-proxy-proxy-agent-86d6967d4f-t7dwt\" (UID: \"8ed49a29-af63-4d51-8729-eb4fcc081daa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6967d4f-t7dwt"
Sep 09 15:21:31.005488 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:31.005325    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrpcw\" (UniqueName: \"kubernetes.io/projected/61925eed-e9ab-4486-b178-9e826d014af4-kube-api-access-hrpcw\") pod \"managed-serviceaccount-addon-agent-77676775cb-ckrdk\" (UID: \"61925eed-e9ab-4486-b178-9e826d014af4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77676775cb-ckrdk"
Sep 09 15:21:31.005488 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:31.005350    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8ed49a29-af63-4d51-8729-eb4fcc081daa-hub\") pod \"cluster-proxy-proxy-agent-86d6967d4f-t7dwt\" (UID: \"8ed49a29-af63-4d51-8729-eb4fcc081daa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6967d4f-t7dwt"
Sep 09 15:21:31.005488 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:31.005366    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8ed49a29-af63-4d51-8729-eb4fcc081daa-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-86d6967d4f-t7dwt\" (UID: \"8ed49a29-af63-4d51-8729-eb4fcc081daa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6967d4f-t7dwt"
Sep 09 15:21:31.005488 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:31.005417    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8ed49a29-af63-4d51-8729-eb4fcc081daa-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-86d6967d4f-t7dwt\" (UID: \"8ed49a29-af63-4d51-8729-eb4fcc081daa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6967d4f-t7dwt"
Sep 09 15:21:31.105849 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:31.105802    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x86j\" (UniqueName: \"kubernetes.io/projected/8ed49a29-af63-4d51-8729-eb4fcc081daa-kube-api-access-9x86j\") pod \"cluster-proxy-proxy-agent-86d6967d4f-t7dwt\" (UID: \"8ed49a29-af63-4d51-8729-eb4fcc081daa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6967d4f-t7dwt"
Sep 09 15:21:31.106007 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:31.105860    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8ed49a29-af63-4d51-8729-eb4fcc081daa-ca\") pod \"cluster-proxy-proxy-agent-86d6967d4f-t7dwt\" (UID: \"8ed49a29-af63-4d51-8729-eb4fcc081daa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6967d4f-t7dwt"
Sep 09 15:21:31.106007 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:31.105892    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrpcw\" (UniqueName: \"kubernetes.io/projected/61925eed-e9ab-4486-b178-9e826d014af4-kube-api-access-hrpcw\") pod \"managed-serviceaccount-addon-agent-77676775cb-ckrdk\" (UID: \"61925eed-e9ab-4486-b178-9e826d014af4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77676775cb-ckrdk"
Sep 09 15:21:31.106007 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:31.105926    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8ed49a29-af63-4d51-8729-eb4fcc081daa-hub\") pod \"cluster-proxy-proxy-agent-86d6967d4f-t7dwt\" (UID: \"8ed49a29-af63-4d51-8729-eb4fcc081daa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6967d4f-t7dwt"
Sep 09 15:21:31.106007 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:31.105948    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8ed49a29-af63-4d51-8729-eb4fcc081daa-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-86d6967d4f-t7dwt\" (UID: \"8ed49a29-af63-4d51-8729-eb4fcc081daa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6967d4f-t7dwt"
Sep 09 15:21:31.106212 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:31.106023    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8ed49a29-af63-4d51-8729-eb4fcc081daa-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-86d6967d4f-t7dwt\" (UID: \"8ed49a29-af63-4d51-8729-eb4fcc081daa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6967d4f-t7dwt"
Sep 09 15:21:31.106212 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:31.106071    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ed49a29-af63-4d51-8729-eb4fcc081daa-additional-service-ca\") pod \"cluster-proxy-proxy-agent-86d6967d4f-t7dwt\" (UID: \"8ed49a29-af63-4d51-8729-eb4fcc081daa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6967d4f-t7dwt"
Sep 09 15:21:31.106212 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:31.106115    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/61925eed-e9ab-4486-b178-9e826d014af4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-77676775cb-ckrdk\" (UID: \"61925eed-e9ab-4486-b178-9e826d014af4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77676775cb-ckrdk"
Sep 09 15:21:31.107077 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:31.107049    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ed49a29-af63-4d51-8729-eb4fcc081daa-additional-service-ca\") pod \"cluster-proxy-proxy-agent-86d6967d4f-t7dwt\" (UID: \"8ed49a29-af63-4d51-8729-eb4fcc081daa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6967d4f-t7dwt"
Sep 09 15:21:31.109461 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:31.109428    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8ed49a29-af63-4d51-8729-eb4fcc081daa-ca\") pod \"cluster-proxy-proxy-agent-86d6967d4f-t7dwt\" (UID: \"8ed49a29-af63-4d51-8729-eb4fcc081daa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6967d4f-t7dwt"
Sep 09 15:21:31.109591 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:31.109570    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8ed49a29-af63-4d51-8729-eb4fcc081daa-hub\") pod \"cluster-proxy-proxy-agent-86d6967d4f-t7dwt\" (UID: \"8ed49a29-af63-4d51-8729-eb4fcc081daa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6967d4f-t7dwt"
Sep 09 15:21:31.109640 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:31.109570    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8ed49a29-af63-4d51-8729-eb4fcc081daa-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-86d6967d4f-t7dwt\" (UID: \"8ed49a29-af63-4d51-8729-eb4fcc081daa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6967d4f-t7dwt"
Sep 09 15:21:31.109674 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:31.109635    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8ed49a29-af63-4d51-8729-eb4fcc081daa-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-86d6967d4f-t7dwt\" (UID: \"8ed49a29-af63-4d51-8729-eb4fcc081daa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6967d4f-t7dwt"
Sep 09 15:21:31.109705 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:31.109677    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/61925eed-e9ab-4486-b178-9e826d014af4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-77676775cb-ckrdk\" (UID: \"61925eed-e9ab-4486-b178-9e826d014af4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77676775cb-ckrdk"
Sep 09 15:21:31.113421 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:31.113392    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrpcw\" (UniqueName: \"kubernetes.io/projected/61925eed-e9ab-4486-b178-9e826d014af4-kube-api-access-hrpcw\") pod \"managed-serviceaccount-addon-agent-77676775cb-ckrdk\" (UID: \"61925eed-e9ab-4486-b178-9e826d014af4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77676775cb-ckrdk"
Sep 09 15:21:31.113421 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:31.113414    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x86j\" (UniqueName: \"kubernetes.io/projected/8ed49a29-af63-4d51-8729-eb4fcc081daa-kube-api-access-9x86j\") pod \"cluster-proxy-proxy-agent-86d6967d4f-t7dwt\" (UID: \"8ed49a29-af63-4d51-8729-eb4fcc081daa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6967d4f-t7dwt"
Sep 09 15:21:31.211035 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:31.210988    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77676775cb-ckrdk"
Sep 09 15:21:31.217360 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:31.217335    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6967d4f-t7dwt"
Sep 09 15:21:31.483365 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:31.483293    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77676775cb-ckrdk"]
Sep 09 15:21:31.486761 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:21:31.486714    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61925eed_e9ab_4486_b178_9e826d014af4.slice/crio-20272c5d063f4606b7a4627ee4f9f49f21a8d364fa32a7498e4a7bee051ee4b4 WatchSource:0}: Error finding container 20272c5d063f4606b7a4627ee4f9f49f21a8d364fa32a7498e4a7bee051ee4b4: Status 404 returned error can't find the container with id 20272c5d063f4606b7a4627ee4f9f49f21a8d364fa32a7498e4a7bee051ee4b4
Sep 09 15:21:31.486988 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:31.486967    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6967d4f-t7dwt"]
Sep 09 15:21:31.489341 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:21:31.489271    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ed49a29_af63_4d51_8729_eb4fcc081daa.slice/crio-550105d8ba023cea9909e528db07a148cc0fa3302e0b2dc3a7f89eeffbb16db2 WatchSource:0}: Error finding container 550105d8ba023cea9909e528db07a148cc0fa3302e0b2dc3a7f89eeffbb16db2: Status 404 returned error can't find the container with id 550105d8ba023cea9909e528db07a148cc0fa3302e0b2dc3a7f89eeffbb16db2
Sep 09 15:21:32.172424 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:32.172381    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77676775cb-ckrdk" event={"ID":"61925eed-e9ab-4486-b178-9e826d014af4","Type":"ContainerStarted","Data":"20272c5d063f4606b7a4627ee4f9f49f21a8d364fa32a7498e4a7bee051ee4b4"}
Sep 09 15:21:32.173490 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:32.173468    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6967d4f-t7dwt" event={"ID":"8ed49a29-af63-4d51-8729-eb4fcc081daa","Type":"ContainerStarted","Data":"550105d8ba023cea9909e528db07a148cc0fa3302e0b2dc3a7f89eeffbb16db2"}
Sep 09 15:21:35.182918 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:35.182873    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77676775cb-ckrdk" event={"ID":"61925eed-e9ab-4486-b178-9e826d014af4","Type":"ContainerStarted","Data":"b59b4bff7d5298a474fb93c735028001cc93ddcfca3b7772c9d9dfb7f8c6241e"}
Sep 09 15:21:35.197768 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:35.197696    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77676775cb-ckrdk" podStartSLOduration=2.230891581 podStartE2EDuration="5.197676441s" totalImagesPullingTime="2.96678486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:21:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:21:35.197248303 +0000 UTC m=+60.796142946" watchObservedRunningTime="2026-09-09 15:21:35.197676441 +0000 UTC m=+60.796571084"
Sep 09 15:21:36.430910 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:36.430882    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-45pwd"
Sep 09 15:21:37.189984 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:37.189943    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6967d4f-t7dwt" event={"ID":"8ed49a29-af63-4d51-8729-eb4fcc081daa","Type":"ContainerStarted","Data":"fa757bdfa297052ac6c540509b07401d1a41046fd3a04d3e90e00a060b429f6e"}
Sep 09 15:21:37.189984 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:37.189992    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6967d4f-t7dwt" event={"ID":"8ed49a29-af63-4d51-8729-eb4fcc081daa","Type":"ContainerStarted","Data":"7041a33bcfd2669d3e32db47325ab170e6ba27fe2261619442faaefe205a8c9b"}
Sep 09 15:21:37.190217 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:37.190005    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6967d4f-t7dwt" event={"ID":"8ed49a29-af63-4d51-8729-eb4fcc081daa","Type":"ContainerStarted","Data":"b81e8ada3a5082e1a83aaedc5715e3ea806ee0d9182ade736c2c71c01994bb5d"}
Sep 09 15:21:37.208859 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:37.208766    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6967d4f-t7dwt" podStartSLOduration=1.7481188300000001 podStartE2EDuration="7.208752293s" totalImagesPullingTime="5.460633463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:21:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:21:37.207250493 +0000 UTC m=+62.806145137" watchObservedRunningTime="2026-09-09 15:21:37.208752293 +0000 UTC m=+62.807646927"
Sep 09 15:21:40.683431 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:40.683393    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26591625-285a-4c93-b5be-7d0f4f0a0aae-original-pull-secret\") pod \"global-pull-secret-syncer-kcl4l\" (UID: \"26591625-285a-4c93-b5be-7d0f4f0a0aae\") " pod="kube-system/global-pull-secret-syncer-kcl4l"
Sep 09 15:21:40.683834 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:40.683454    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57473bc0-f6d0-45f1-8954-93a7d2f4321b-metrics-certs\") pod \"network-metrics-daemon-v5kbj\" (UID: \"57473bc0-f6d0-45f1-8954-93a7d2f4321b\") " pod="openshift-multus/network-metrics-daemon-v5kbj"
Sep 09 15:21:40.686246 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:40.686229    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 15:21:40.686308 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:40.686269    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 15:21:40.694409 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:21:40.694380    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 15:21:40.694512 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:21:40.694457    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57473bc0-f6d0-45f1-8954-93a7d2f4321b-metrics-certs podName:57473bc0-f6d0-45f1-8954-93a7d2f4321b nodeName:}" failed. No retries permitted until 2026-09-09 15:22:44.694439119 +0000 UTC m=+130.293333741 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57473bc0-f6d0-45f1-8954-93a7d2f4321b-metrics-certs") pod "network-metrics-daemon-v5kbj" (UID: "57473bc0-f6d0-45f1-8954-93a7d2f4321b") : secret "metrics-daemon-secret" not found
Sep 09 15:21:40.696484 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:40.696461    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26591625-285a-4c93-b5be-7d0f4f0a0aae-original-pull-secret\") pod \"global-pull-secret-syncer-kcl4l\" (UID: \"26591625-285a-4c93-b5be-7d0f4f0a0aae\") " pod="kube-system/global-pull-secret-syncer-kcl4l"
Sep 09 15:21:40.784000 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:40.783948    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll8rs\" (UniqueName: \"kubernetes.io/projected/d830519d-a525-4029-ae78-d30bb04e19e5-kube-api-access-ll8rs\") pod \"network-check-target-xwx7m\" (UID: \"d830519d-a525-4029-ae78-d30bb04e19e5\") " pod="openshift-network-diagnostics/network-check-target-xwx7m"
Sep 09 15:21:40.786809 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:40.786788    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 15:21:40.796903 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:40.796885    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 15:21:40.808115 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:40.808088    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll8rs\" (UniqueName: \"kubernetes.io/projected/d830519d-a525-4029-ae78-d30bb04e19e5-kube-api-access-ll8rs\") pod \"network-check-target-xwx7m\" (UID: \"d830519d-a525-4029-ae78-d30bb04e19e5\") " pod="openshift-network-diagnostics/network-check-target-xwx7m"
Sep 09 15:21:40.869470 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:40.869423    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcl4l"
Sep 09 15:21:40.971844 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:40.971767    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7889l\""
Sep 09 15:21:40.979798 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:40.979778    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwx7m"
Sep 09 15:21:40.986238 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:40.986213    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acc8de36-2e42-4471-a31e-38c66f87ab8c-metrics-tls\") pod \"dns-default-cwkld\" (UID: \"acc8de36-2e42-4471-a31e-38c66f87ab8c\") " pod="openshift-dns/dns-default-cwkld"
Sep 09 15:21:40.986341 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:21:40.986315    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 15:21:40.986376 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:21:40.986370    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acc8de36-2e42-4471-a31e-38c66f87ab8c-metrics-tls podName:acc8de36-2e42-4471-a31e-38c66f87ab8c nodeName:}" failed. No retries permitted until 2026-09-09 15:22:12.986351576 +0000 UTC m=+98.585246216 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acc8de36-2e42-4471-a31e-38c66f87ab8c-metrics-tls") pod "dns-default-cwkld" (UID: "acc8de36-2e42-4471-a31e-38c66f87ab8c") : secret "dns-default-metrics-tls" not found
Sep 09 15:21:41.134924 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:41.134895    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kcl4l"]
Sep 09 15:21:41.137640 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:21:41.137615    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26591625_285a_4c93_b5be_7d0f4f0a0aae.slice/crio-cd6899800c28aa17945aa8e37b4a49d5be5136002943f075286fa685237befc8 WatchSource:0}: Error finding container cd6899800c28aa17945aa8e37b4a49d5be5136002943f075286fa685237befc8: Status 404 returned error can't find the container with id cd6899800c28aa17945aa8e37b4a49d5be5136002943f075286fa685237befc8
Sep 09 15:21:41.200886 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:41.200856    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kcl4l" event={"ID":"26591625-285a-4c93-b5be-7d0f4f0a0aae","Type":"ContainerStarted","Data":"cd6899800c28aa17945aa8e37b4a49d5be5136002943f075286fa685237befc8"}
Sep 09 15:21:41.247156 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:21:41.247123    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd830519d_a525_4029_ae78_d30bb04e19e5.slice/crio-0404329e2d1a5e9fb9763aba61f6935caf6b336d18a48af85b4ffa7b399ce2a4 WatchSource:0}: Error finding container 0404329e2d1a5e9fb9763aba61f6935caf6b336d18a48af85b4ffa7b399ce2a4: Status 404 returned error can't find the container with id 0404329e2d1a5e9fb9763aba61f6935caf6b336d18a48af85b4ffa7b399ce2a4
Sep 09 15:21:41.247844 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:41.247821    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xwx7m"]
Sep 09 15:21:42.205497 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:42.205462    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xwx7m" event={"ID":"d830519d-a525-4029-ae78-d30bb04e19e5","Type":"ContainerStarted","Data":"0404329e2d1a5e9fb9763aba61f6935caf6b336d18a48af85b4ffa7b399ce2a4"}
Sep 09 15:21:47.222135 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:47.222097    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kcl4l" event={"ID":"26591625-285a-4c93-b5be-7d0f4f0a0aae","Type":"ContainerStarted","Data":"bd61bea5212157eddc6a9efb81d7fe05651ce5bf95a1bc662a167ac3fd256013"}
Sep 09 15:21:47.223552 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:47.223523    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xwx7m" event={"ID":"d830519d-a525-4029-ae78-d30bb04e19e5","Type":"ContainerStarted","Data":"4a2c5987243911a65da0dbcc643020663457ced97e389c5ee89fe46748c7918e"}
Sep 09 15:21:47.223684 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:47.223665    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xwx7m"
Sep 09 15:21:47.239053 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:47.237135    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kcl4l" podStartSLOduration=67.030538982 podStartE2EDuration="1m12.237117333s" totalImagesPullingTime="5.206578351s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:20:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:21:47.235576076 +0000 UTC m=+72.834470723" watchObservedRunningTime="2026-09-09 15:21:47.237117333 +0000 UTC m=+72.836011979"
Sep 09 15:21:47.248849 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:21:47.248813    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xwx7m" podStartSLOduration=67.163412952 podStartE2EDuration="1m12.248801102s" totalImagesPullingTime="5.08538815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:20:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:21:47.248365923 +0000 UTC m=+72.847260567" watchObservedRunningTime="2026-09-09 15:21:47.248801102 +0000 UTC m=+72.847695739"
Sep 09 15:22:13.014561 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:22:13.014420    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acc8de36-2e42-4471-a31e-38c66f87ab8c-metrics-tls\") pod \"dns-default-cwkld\" (UID: \"acc8de36-2e42-4471-a31e-38c66f87ab8c\") " pod="openshift-dns/dns-default-cwkld"
Sep 09 15:22:13.014561 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:22:13.014565    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 15:22:13.015103 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:22:13.014637    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acc8de36-2e42-4471-a31e-38c66f87ab8c-metrics-tls podName:acc8de36-2e42-4471-a31e-38c66f87ab8c nodeName:}" failed. No retries permitted until 2026-09-09 15:23:17.014620957 +0000 UTC m=+162.613515601 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acc8de36-2e42-4471-a31e-38c66f87ab8c-metrics-tls") pod "dns-default-cwkld" (UID: "acc8de36-2e42-4471-a31e-38c66f87ab8c") : secret "dns-default-metrics-tls" not found
Sep 09 15:22:18.229381 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:22:18.229352    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xwx7m"
Sep 09 15:22:44.721046 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:22:44.721002    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57473bc0-f6d0-45f1-8954-93a7d2f4321b-metrics-certs\") pod \"network-metrics-daemon-v5kbj\" (UID: \"57473bc0-f6d0-45f1-8954-93a7d2f4321b\") " pod="openshift-multus/network-metrics-daemon-v5kbj"
Sep 09 15:22:44.721610 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:22:44.721160    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 15:22:44.721610 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:22:44.721248    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57473bc0-f6d0-45f1-8954-93a7d2f4321b-metrics-certs podName:57473bc0-f6d0-45f1-8954-93a7d2f4321b nodeName:}" failed. No retries permitted until 2026-09-09 15:24:46.721225648 +0000 UTC m=+252.320120272 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57473bc0-f6d0-45f1-8954-93a7d2f4321b-metrics-certs") pod "network-metrics-daemon-v5kbj" (UID: "57473bc0-f6d0-45f1-8954-93a7d2f4321b") : secret "metrics-daemon-secret" not found
Sep 09 15:23:12.163589 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:23:12.163542    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-cwkld" podUID="acc8de36-2e42-4471-a31e-38c66f87ab8c"
Sep 09 15:23:12.467098 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:23:12.467065    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cwkld"
Sep 09 15:23:12.976043 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:23:12.976005    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-v5kbj" podUID="57473bc0-f6d0-45f1-8954-93a7d2f4321b"
Sep 09 15:23:17.034024 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:23:17.033989    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acc8de36-2e42-4471-a31e-38c66f87ab8c-metrics-tls\") pod \"dns-default-cwkld\" (UID: \"acc8de36-2e42-4471-a31e-38c66f87ab8c\") " pod="openshift-dns/dns-default-cwkld"
Sep 09 15:23:17.034475 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:23:17.034103    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 15:23:17.034475 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:23:17.034166    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acc8de36-2e42-4471-a31e-38c66f87ab8c-metrics-tls podName:acc8de36-2e42-4471-a31e-38c66f87ab8c nodeName:}" failed. No retries permitted until 2026-09-09 15:25:19.034152319 +0000 UTC m=+284.633046941 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acc8de36-2e42-4471-a31e-38c66f87ab8c-metrics-tls") pod "dns-default-cwkld" (UID: "acc8de36-2e42-4471-a31e-38c66f87ab8c") : secret "dns-default-metrics-tls" not found
Sep 09 15:23:25.955854 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:23:25.955813    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5kbj"
Sep 09 15:23:35.536830 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:23:35.536789    2553 generic.go:356] "Generic (PLEG): container finished" podID="61925eed-e9ab-4486-b178-9e826d014af4" containerID="b59b4bff7d5298a474fb93c735028001cc93ddcfca3b7772c9d9dfb7f8c6241e" exitCode=255
Sep 09 15:23:35.536830 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:23:35.536828    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77676775cb-ckrdk" event={"ID":"61925eed-e9ab-4486-b178-9e826d014af4","Type":"ContainerDied","Data":"b59b4bff7d5298a474fb93c735028001cc93ddcfca3b7772c9d9dfb7f8c6241e"}
Sep 09 15:23:35.537274 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:23:35.537087    2553 scope.go:117] "RemoveContainer" containerID="b59b4bff7d5298a474fb93c735028001cc93ddcfca3b7772c9d9dfb7f8c6241e"
Sep 09 15:23:36.541567 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:23:36.541532    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77676775cb-ckrdk" event={"ID":"61925eed-e9ab-4486-b178-9e826d014af4","Type":"ContainerStarted","Data":"b472cf85628028873284fd049f817b92d4e9f6e5c55962a088b4f7d68222f422"}
Sep 09 15:24:15.637702 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:15.637670    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hz29k"]
Sep 09 15:24:15.639548 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:15.639529    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hz29k"
Sep 09 15:24:15.642042 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:15.642024    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 15:24:15.642150 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:15.642042    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 15:24:15.642233 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:15.642218    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6tbnq\""
Sep 09 15:24:15.643190 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:15.643170    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 15:24:15.643284 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:15.643229    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 15:24:15.650153 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:15.650132    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hz29k"]
Sep 09 15:24:15.740063 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:15.740023    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hz29k\" (UID: \"f3f904f8-3a34-41d2-8f0c-0695ad8b5209\") " pod="openshift-insights/insights-runtime-extractor-hz29k"
Sep 09 15:24:15.740221 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:15.740092    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hz29k\" (UID: \"f3f904f8-3a34-41d2-8f0c-0695ad8b5209\") " pod="openshift-insights/insights-runtime-extractor-hz29k"
Sep 09 15:24:15.740221 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:15.740131    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g9tb\" (UniqueName: \"kubernetes.io/projected/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-kube-api-access-2g9tb\") pod \"insights-runtime-extractor-hz29k\" (UID: \"f3f904f8-3a34-41d2-8f0c-0695ad8b5209\") " pod="openshift-insights/insights-runtime-extractor-hz29k"
Sep 09 15:24:15.740221 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:15.740169    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-crio-socket\") pod \"insights-runtime-extractor-hz29k\" (UID: \"f3f904f8-3a34-41d2-8f0c-0695ad8b5209\") " pod="openshift-insights/insights-runtime-extractor-hz29k"
Sep 09 15:24:15.740221 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:15.740189    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-data-volume\") pod \"insights-runtime-extractor-hz29k\" (UID: \"f3f904f8-3a34-41d2-8f0c-0695ad8b5209\") " pod="openshift-insights/insights-runtime-extractor-hz29k"
Sep 09 15:24:15.840701 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:15.840660    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-crio-socket\") pod \"insights-runtime-extractor-hz29k\" (UID: \"f3f904f8-3a34-41d2-8f0c-0695ad8b5209\") " pod="openshift-insights/insights-runtime-extractor-hz29k"
Sep 09 15:24:15.840701 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:15.840706    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-data-volume\") pod \"insights-runtime-extractor-hz29k\" (UID: \"f3f904f8-3a34-41d2-8f0c-0695ad8b5209\") " pod="openshift-insights/insights-runtime-extractor-hz29k"
Sep 09 15:24:15.840972 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:15.840769    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hz29k\" (UID: \"f3f904f8-3a34-41d2-8f0c-0695ad8b5209\") " pod="openshift-insights/insights-runtime-extractor-hz29k"
Sep 09 15:24:15.840972 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:15.840816    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hz29k\" (UID: \"f3f904f8-3a34-41d2-8f0c-0695ad8b5209\") " pod="openshift-insights/insights-runtime-extractor-hz29k"
Sep 09 15:24:15.840972 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:15.840821    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-crio-socket\") pod \"insights-runtime-extractor-hz29k\" (UID: \"f3f904f8-3a34-41d2-8f0c-0695ad8b5209\") " pod="openshift-insights/insights-runtime-extractor-hz29k"
Sep 09 15:24:15.840972 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:15.840850    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g9tb\" (UniqueName: \"kubernetes.io/projected/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-kube-api-access-2g9tb\") pod \"insights-runtime-extractor-hz29k\" (UID: \"f3f904f8-3a34-41d2-8f0c-0695ad8b5209\") " pod="openshift-insights/insights-runtime-extractor-hz29k"
Sep 09 15:24:15.840972 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:24:15.840931    2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 15:24:15.840972 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:24:15.840945    2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 15:24:15.841239 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:24:15.841012    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-insights-runtime-extractor-tls podName:f3f904f8-3a34-41d2-8f0c-0695ad8b5209 nodeName:}" failed. No retries permitted until 2026-09-09 15:24:16.340996874 +0000 UTC m=+221.939891500 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hz29k" (UID: "f3f904f8-3a34-41d2-8f0c-0695ad8b5209") : secret "insights-runtime-extractor-tls" not found
Sep 09 15:24:15.841239 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:24:15.841028    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-kube-rbac-proxy-cm podName:f3f904f8-3a34-41d2-8f0c-0695ad8b5209 nodeName:}" failed. No retries permitted until 2026-09-09 15:24:16.341020846 +0000 UTC m=+221.939915468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hz29k" (UID: "f3f904f8-3a34-41d2-8f0c-0695ad8b5209") : configmap "kube-rbac-proxy" not found
Sep 09 15:24:15.841239 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:15.841141    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-data-volume\") pod \"insights-runtime-extractor-hz29k\" (UID: \"f3f904f8-3a34-41d2-8f0c-0695ad8b5209\") " pod="openshift-insights/insights-runtime-extractor-hz29k"
Sep 09 15:24:15.851137 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:15.851114    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g9tb\" (UniqueName: \"kubernetes.io/projected/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-kube-api-access-2g9tb\") pod \"insights-runtime-extractor-hz29k\" (UID: \"f3f904f8-3a34-41d2-8f0c-0695ad8b5209\") " pod="openshift-insights/insights-runtime-extractor-hz29k"
Sep 09 15:24:16.343985 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:16.343940    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hz29k\" (UID: \"f3f904f8-3a34-41d2-8f0c-0695ad8b5209\") " pod="openshift-insights/insights-runtime-extractor-hz29k"
Sep 09 15:24:16.343985 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:16.344004    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hz29k\" (UID: \"f3f904f8-3a34-41d2-8f0c-0695ad8b5209\") " pod="openshift-insights/insights-runtime-extractor-hz29k"
Sep 09 15:24:16.344230 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:24:16.344087    2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 15:24:16.344230 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:24:16.344102    2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 15:24:16.344230 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:24:16.344154    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-kube-rbac-proxy-cm podName:f3f904f8-3a34-41d2-8f0c-0695ad8b5209 nodeName:}" failed. No retries permitted until 2026-09-09 15:24:17.344137697 +0000 UTC m=+222.943032319 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hz29k" (UID: "f3f904f8-3a34-41d2-8f0c-0695ad8b5209") : configmap "kube-rbac-proxy" not found
Sep 09 15:24:16.344230 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:24:16.344166    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-insights-runtime-extractor-tls podName:f3f904f8-3a34-41d2-8f0c-0695ad8b5209 nodeName:}" failed. No retries permitted until 2026-09-09 15:24:17.344160662 +0000 UTC m=+222.943055284 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hz29k" (UID: "f3f904f8-3a34-41d2-8f0c-0695ad8b5209") : secret "insights-runtime-extractor-tls" not found
Sep 09 15:24:17.352820 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:17.352780    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hz29k\" (UID: \"f3f904f8-3a34-41d2-8f0c-0695ad8b5209\") " pod="openshift-insights/insights-runtime-extractor-hz29k"
Sep 09 15:24:17.353210 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:17.352834    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hz29k\" (UID: \"f3f904f8-3a34-41d2-8f0c-0695ad8b5209\") " pod="openshift-insights/insights-runtime-extractor-hz29k"
Sep 09 15:24:17.353210 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:24:17.352926    2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 15:24:17.353210 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:24:17.352934    2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 15:24:17.353210 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:24:17.352984    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-kube-rbac-proxy-cm podName:f3f904f8-3a34-41d2-8f0c-0695ad8b5209 nodeName:}" failed. No retries permitted until 2026-09-09 15:24:19.352968204 +0000 UTC m=+224.951862828 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hz29k" (UID: "f3f904f8-3a34-41d2-8f0c-0695ad8b5209") : configmap "kube-rbac-proxy" not found
Sep 09 15:24:17.353210 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:24:17.352998    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-insights-runtime-extractor-tls podName:f3f904f8-3a34-41d2-8f0c-0695ad8b5209 nodeName:}" failed. No retries permitted until 2026-09-09 15:24:19.352990982 +0000 UTC m=+224.951885604 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hz29k" (UID: "f3f904f8-3a34-41d2-8f0c-0695ad8b5209") : secret "insights-runtime-extractor-tls" not found
Sep 09 15:24:17.798550 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:17.798522    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wgm9j_3cfc8b11-7c62-46fc-8a5e-d5d889501cfd/dns-node-resolver/0.log"
Sep 09 15:24:18.310338 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:18.310299    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-54wmr"]
Sep 09 15:24:18.312467 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:18.312448    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-54wmr"
Sep 09 15:24:18.314948 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:18.314927    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-28jbz\""
Sep 09 15:24:18.315083 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:18.314925    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 09 15:24:18.315083 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:18.314928    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 09 15:24:18.316001 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:18.315986    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 09 15:24:18.316096 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:18.316047    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 09 15:24:18.321218 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:18.321197    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-54wmr"]
Sep 09 15:24:18.361491 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:18.361454    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c836d682-2882-4e60-8adc-cdde616f4872-signing-cabundle\") pod \"service-ca-75d99c484-54wmr\" (UID: \"c836d682-2882-4e60-8adc-cdde616f4872\") " pod="openshift-service-ca/service-ca-75d99c484-54wmr"
Sep 09 15:24:18.361924 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:18.361501    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c836d682-2882-4e60-8adc-cdde616f4872-signing-key\") pod \"service-ca-75d99c484-54wmr\" (UID: \"c836d682-2882-4e60-8adc-cdde616f4872\") " pod="openshift-service-ca/service-ca-75d99c484-54wmr"
Sep 09 15:24:18.361924 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:18.361560    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7klqf\" (UniqueName: \"kubernetes.io/projected/c836d682-2882-4e60-8adc-cdde616f4872-kube-api-access-7klqf\") pod \"service-ca-75d99c484-54wmr\" (UID: \"c836d682-2882-4e60-8adc-cdde616f4872\") " pod="openshift-service-ca/service-ca-75d99c484-54wmr"
Sep 09 15:24:18.462403 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:18.462367    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7klqf\" (UniqueName: \"kubernetes.io/projected/c836d682-2882-4e60-8adc-cdde616f4872-kube-api-access-7klqf\") pod \"service-ca-75d99c484-54wmr\" (UID: \"c836d682-2882-4e60-8adc-cdde616f4872\") " pod="openshift-service-ca/service-ca-75d99c484-54wmr"
Sep 09 15:24:18.462403 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:18.462413    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c836d682-2882-4e60-8adc-cdde616f4872-signing-cabundle\") pod \"service-ca-75d99c484-54wmr\" (UID: \"c836d682-2882-4e60-8adc-cdde616f4872\") " pod="openshift-service-ca/service-ca-75d99c484-54wmr"
Sep 09 15:24:18.462597 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:18.462447    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c836d682-2882-4e60-8adc-cdde616f4872-signing-key\") pod \"service-ca-75d99c484-54wmr\" (UID: \"c836d682-2882-4e60-8adc-cdde616f4872\") " pod="openshift-service-ca/service-ca-75d99c484-54wmr"
Sep 09 15:24:18.463159 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:18.463130    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c836d682-2882-4e60-8adc-cdde616f4872-signing-cabundle\") pod \"service-ca-75d99c484-54wmr\" (UID: \"c836d682-2882-4e60-8adc-cdde616f4872\") " pod="openshift-service-ca/service-ca-75d99c484-54wmr"
Sep 09 15:24:18.464908 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:18.464879    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c836d682-2882-4e60-8adc-cdde616f4872-signing-key\") pod \"service-ca-75d99c484-54wmr\" (UID: \"c836d682-2882-4e60-8adc-cdde616f4872\") " pod="openshift-service-ca/service-ca-75d99c484-54wmr"
Sep 09 15:24:18.472438 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:18.472412    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7klqf\" (UniqueName: \"kubernetes.io/projected/c836d682-2882-4e60-8adc-cdde616f4872-kube-api-access-7klqf\") pod \"service-ca-75d99c484-54wmr\" (UID: \"c836d682-2882-4e60-8adc-cdde616f4872\") " pod="openshift-service-ca/service-ca-75d99c484-54wmr"
Sep 09 15:24:18.621800 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:18.621693    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-54wmr"
Sep 09 15:24:18.799912 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:18.799882    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-txxcj_1aaf71f6-9195-4b50-a9a4-3cb85b10956a/node-ca/0.log"
Sep 09 15:24:18.889943 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:18.889867    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-54wmr"]
Sep 09 15:24:18.892981 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:24:18.892954    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc836d682_2882_4e60_8adc_cdde616f4872.slice/crio-6bfeb03f3d44eb417bfa6c8236902e214bea001e32fb6051c46a18285f87ac34 WatchSource:0}: Error finding container 6bfeb03f3d44eb417bfa6c8236902e214bea001e32fb6051c46a18285f87ac34: Status 404 returned error can't find the container with id 6bfeb03f3d44eb417bfa6c8236902e214bea001e32fb6051c46a18285f87ac34
Sep 09 15:24:19.370108 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:19.370068    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hz29k\" (UID: \"f3f904f8-3a34-41d2-8f0c-0695ad8b5209\") " pod="openshift-insights/insights-runtime-extractor-hz29k"
Sep 09 15:24:19.370509 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:19.370133    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hz29k\" (UID: \"f3f904f8-3a34-41d2-8f0c-0695ad8b5209\") " pod="openshift-insights/insights-runtime-extractor-hz29k"
Sep 09 15:24:19.370509 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:24:19.370213    2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 15:24:19.370509 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:24:19.370229    2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 15:24:19.370509 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:24:19.370268    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-kube-rbac-proxy-cm podName:f3f904f8-3a34-41d2-8f0c-0695ad8b5209 nodeName:}" failed. No retries permitted until 2026-09-09 15:24:23.370252828 +0000 UTC m=+228.969147450 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hz29k" (UID: "f3f904f8-3a34-41d2-8f0c-0695ad8b5209") : configmap "kube-rbac-proxy" not found
Sep 09 15:24:19.370509 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:24:19.370287    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-insights-runtime-extractor-tls podName:f3f904f8-3a34-41d2-8f0c-0695ad8b5209 nodeName:}" failed. No retries permitted until 2026-09-09 15:24:23.370273621 +0000 UTC m=+228.969168243 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hz29k" (UID: "f3f904f8-3a34-41d2-8f0c-0695ad8b5209") : secret "insights-runtime-extractor-tls" not found
Sep 09 15:24:19.402349 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:19.402319    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7kz9s_d2082125-0643-4b68-a968-a45d7d8aac36/serve-healthcheck-canary/0.log"
Sep 09 15:24:19.668210 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:19.668123    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-54wmr" event={"ID":"c836d682-2882-4e60-8adc-cdde616f4872","Type":"ContainerStarted","Data":"6bfeb03f3d44eb417bfa6c8236902e214bea001e32fb6051c46a18285f87ac34"}
Sep 09 15:24:21.677222 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:21.677183    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-54wmr" event={"ID":"c836d682-2882-4e60-8adc-cdde616f4872","Type":"ContainerStarted","Data":"b8e80fdd116a8109b49d8c14e259966e38a04d1212cf83f203be48fbe3ea2cef"}
Sep 09 15:24:21.692036 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:21.691987    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-54wmr" podStartSLOduration=1.684512126 podStartE2EDuration="3.691965319s" totalImagesPullingTime="2.007453193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:24:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:24:21.691326255 +0000 UTC m=+227.290220898" watchObservedRunningTime="2026-09-09 15:24:21.691965319 +0000 UTC m=+227.290860007"
Sep 09 15:24:23.398012 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:23.397951    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hz29k\" (UID: \"f3f904f8-3a34-41d2-8f0c-0695ad8b5209\") " pod="openshift-insights/insights-runtime-extractor-hz29k"
Sep 09 15:24:23.398012 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:23.398022    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hz29k\" (UID: \"f3f904f8-3a34-41d2-8f0c-0695ad8b5209\") " pod="openshift-insights/insights-runtime-extractor-hz29k"
Sep 09 15:24:23.398494 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:24:23.398117    2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 15:24:23.398494 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:24:23.398189    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-insights-runtime-extractor-tls podName:f3f904f8-3a34-41d2-8f0c-0695ad8b5209 nodeName:}" failed. No retries permitted until 2026-09-09 15:24:31.398171827 +0000 UTC m=+236.997066454 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hz29k" (UID: "f3f904f8-3a34-41d2-8f0c-0695ad8b5209") : secret "insights-runtime-extractor-tls" not found
Sep 09 15:24:23.398494 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:24:23.398117    2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 15:24:23.398494 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:24:23.398259    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-kube-rbac-proxy-cm podName:f3f904f8-3a34-41d2-8f0c-0695ad8b5209 nodeName:}" failed. No retries permitted until 2026-09-09 15:24:31.398246792 +0000 UTC m=+236.997141414 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hz29k" (UID: "f3f904f8-3a34-41d2-8f0c-0695ad8b5209") : configmap "kube-rbac-proxy" not found
Sep 09 15:24:31.461081 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:31.461041    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hz29k\" (UID: \"f3f904f8-3a34-41d2-8f0c-0695ad8b5209\") " pod="openshift-insights/insights-runtime-extractor-hz29k"
Sep 09 15:24:31.461463 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:31.461096    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hz29k\" (UID: \"f3f904f8-3a34-41d2-8f0c-0695ad8b5209\") " pod="openshift-insights/insights-runtime-extractor-hz29k"
Sep 09 15:24:31.461463 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:24:31.461233    2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 15:24:31.461463 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:24:31.461291    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-insights-runtime-extractor-tls podName:f3f904f8-3a34-41d2-8f0c-0695ad8b5209 nodeName:}" failed. No retries permitted until 2026-09-09 15:24:47.46127671 +0000 UTC m=+253.060171332 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hz29k" (UID: "f3f904f8-3a34-41d2-8f0c-0695ad8b5209") : secret "insights-runtime-extractor-tls" not found
Sep 09 15:24:31.461570 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:31.461524    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hz29k\" (UID: \"f3f904f8-3a34-41d2-8f0c-0695ad8b5209\") " pod="openshift-insights/insights-runtime-extractor-hz29k"
Sep 09 15:24:37.326480 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:37.326445    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-d78b99cdd-bhxqb"]
Sep 09 15:24:37.329140 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:37.329123    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d78b99cdd-bhxqb"
Sep 09 15:24:37.331424 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:37.331399    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 15:24:37.331565 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:37.331528    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 15:24:37.331726 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:37.331709    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 15:24:37.332699 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:37.332683    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pzhsz\""
Sep 09 15:24:37.337566 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:37.337548    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 15:24:37.340834 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:37.340813    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d78b99cdd-bhxqb"]
Sep 09 15:24:37.407550 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:37.407519    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d297493c-df72-413e-988f-c4d38b6bbf5f-bound-sa-token\") pod \"image-registry-d78b99cdd-bhxqb\" (UID: \"d297493c-df72-413e-988f-c4d38b6bbf5f\") " pod="openshift-image-registry/image-registry-d78b99cdd-bhxqb"
Sep 09 15:24:37.407550 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:37.407554    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d297493c-df72-413e-988f-c4d38b6bbf5f-trusted-ca\") pod \"image-registry-d78b99cdd-bhxqb\" (UID: \"d297493c-df72-413e-988f-c4d38b6bbf5f\") " pod="openshift-image-registry/image-registry-d78b99cdd-bhxqb"
Sep 09 15:24:37.407796 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:37.407590    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d297493c-df72-413e-988f-c4d38b6bbf5f-registry-tls\") pod \"image-registry-d78b99cdd-bhxqb\" (UID: \"d297493c-df72-413e-988f-c4d38b6bbf5f\") " pod="openshift-image-registry/image-registry-d78b99cdd-bhxqb"
Sep 09 15:24:37.407796 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:37.407695    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d297493c-df72-413e-988f-c4d38b6bbf5f-installation-pull-secrets\") pod \"image-registry-d78b99cdd-bhxqb\" (UID: \"d297493c-df72-413e-988f-c4d38b6bbf5f\") " pod="openshift-image-registry/image-registry-d78b99cdd-bhxqb"
Sep 09 15:24:37.407796 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:37.407745    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d297493c-df72-413e-988f-c4d38b6bbf5f-image-registry-private-configuration\") pod \"image-registry-d78b99cdd-bhxqb\" (UID: \"d297493c-df72-413e-988f-c4d38b6bbf5f\") " pod="openshift-image-registry/image-registry-d78b99cdd-bhxqb"
Sep 09 15:24:37.407796 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:37.407779    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d297493c-df72-413e-988f-c4d38b6bbf5f-registry-certificates\") pod \"image-registry-d78b99cdd-bhxqb\" (UID: \"d297493c-df72-413e-988f-c4d38b6bbf5f\") " pod="openshift-image-registry/image-registry-d78b99cdd-bhxqb"
Sep 09 15:24:37.407920 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:37.407818    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d297493c-df72-413e-988f-c4d38b6bbf5f-ca-trust-extracted\") pod \"image-registry-d78b99cdd-bhxqb\" (UID: \"d297493c-df72-413e-988f-c4d38b6bbf5f\") " pod="openshift-image-registry/image-registry-d78b99cdd-bhxqb"
Sep 09 15:24:37.407920 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:37.407841    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t266c\" (UniqueName: \"kubernetes.io/projected/d297493c-df72-413e-988f-c4d38b6bbf5f-kube-api-access-t266c\") pod \"image-registry-d78b99cdd-bhxqb\" (UID: \"d297493c-df72-413e-988f-c4d38b6bbf5f\") " pod="openshift-image-registry/image-registry-d78b99cdd-bhxqb"
Sep 09 15:24:37.508359 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:37.508321    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d297493c-df72-413e-988f-c4d38b6bbf5f-trusted-ca\") pod \"image-registry-d78b99cdd-bhxqb\" (UID: \"d297493c-df72-413e-988f-c4d38b6bbf5f\") " pod="openshift-image-registry/image-registry-d78b99cdd-bhxqb"
Sep 09 15:24:37.508359 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:37.508363    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d297493c-df72-413e-988f-c4d38b6bbf5f-registry-tls\") pod \"image-registry-d78b99cdd-bhxqb\" (UID: \"d297493c-df72-413e-988f-c4d38b6bbf5f\") " pod="openshift-image-registry/image-registry-d78b99cdd-bhxqb"
Sep 09 15:24:37.508570 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:37.508507    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d297493c-df72-413e-988f-c4d38b6bbf5f-installation-pull-secrets\") pod \"image-registry-d78b99cdd-bhxqb\" (UID: \"d297493c-df72-413e-988f-c4d38b6bbf5f\") " pod="openshift-image-registry/image-registry-d78b99cdd-bhxqb"
Sep 09 15:24:37.508570 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:37.508538    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d297493c-df72-413e-988f-c4d38b6bbf5f-image-registry-private-configuration\") pod \"image-registry-d78b99cdd-bhxqb\" (UID: \"d297493c-df72-413e-988f-c4d38b6bbf5f\") " pod="openshift-image-registry/image-registry-d78b99cdd-bhxqb"
Sep 09 15:24:37.508570 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:37.508569    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d297493c-df72-413e-988f-c4d38b6bbf5f-registry-certificates\") pod \"image-registry-d78b99cdd-bhxqb\" (UID: \"d297493c-df72-413e-988f-c4d38b6bbf5f\") " pod="openshift-image-registry/image-registry-d78b99cdd-bhxqb"
Sep 09 15:24:37.508718 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:37.508599    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d297493c-df72-413e-988f-c4d38b6bbf5f-ca-trust-extracted\") pod \"image-registry-d78b99cdd-bhxqb\" (UID: \"d297493c-df72-413e-988f-c4d38b6bbf5f\") " pod="openshift-image-registry/image-registry-d78b99cdd-bhxqb"
Sep 09 15:24:37.508801 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:37.508710    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t266c\" (UniqueName: \"kubernetes.io/projected/d297493c-df72-413e-988f-c4d38b6bbf5f-kube-api-access-t266c\") pod \"image-registry-d78b99cdd-bhxqb\" (UID: \"d297493c-df72-413e-988f-c4d38b6bbf5f\") " pod="openshift-image-registry/image-registry-d78b99cdd-bhxqb"
Sep 09 15:24:37.508855 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:37.508801    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d297493c-df72-413e-988f-c4d38b6bbf5f-bound-sa-token\") pod \"image-registry-d78b99cdd-bhxqb\" (UID: \"d297493c-df72-413e-988f-c4d38b6bbf5f\") " pod="openshift-image-registry/image-registry-d78b99cdd-bhxqb"
Sep 09 15:24:37.509270 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:37.509239    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d297493c-df72-413e-988f-c4d38b6bbf5f-ca-trust-extracted\") pod \"image-registry-d78b99cdd-bhxqb\" (UID: \"d297493c-df72-413e-988f-c4d38b6bbf5f\") " pod="openshift-image-registry/image-registry-d78b99cdd-bhxqb"
Sep 09 15:24:37.509454 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:37.509415    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d297493c-df72-413e-988f-c4d38b6bbf5f-registry-certificates\") pod \"image-registry-d78b99cdd-bhxqb\" (UID: \"d297493c-df72-413e-988f-c4d38b6bbf5f\") " pod="openshift-image-registry/image-registry-d78b99cdd-bhxqb"
Sep 09 15:24:37.509557 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:37.509481    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d297493c-df72-413e-988f-c4d38b6bbf5f-trusted-ca\") pod \"image-registry-d78b99cdd-bhxqb\" (UID: \"d297493c-df72-413e-988f-c4d38b6bbf5f\") " pod="openshift-image-registry/image-registry-d78b99cdd-bhxqb"
Sep 09 15:24:37.511012 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:37.510993    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d297493c-df72-413e-988f-c4d38b6bbf5f-registry-tls\") pod \"image-registry-d78b99cdd-bhxqb\" (UID: \"d297493c-df72-413e-988f-c4d38b6bbf5f\") " pod="openshift-image-registry/image-registry-d78b99cdd-bhxqb"
Sep 09 15:24:37.511416 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:37.511389    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d297493c-df72-413e-988f-c4d38b6bbf5f-installation-pull-secrets\") pod \"image-registry-d78b99cdd-bhxqb\" (UID: \"d297493c-df72-413e-988f-c4d38b6bbf5f\") " pod="openshift-image-registry/image-registry-d78b99cdd-bhxqb"
Sep 09 15:24:37.511501 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:37.511469    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d297493c-df72-413e-988f-c4d38b6bbf5f-image-registry-private-configuration\") pod \"image-registry-d78b99cdd-bhxqb\" (UID: \"d297493c-df72-413e-988f-c4d38b6bbf5f\") " pod="openshift-image-registry/image-registry-d78b99cdd-bhxqb"
Sep 09 15:24:37.518376 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:37.518355    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t266c\" (UniqueName: \"kubernetes.io/projected/d297493c-df72-413e-988f-c4d38b6bbf5f-kube-api-access-t266c\") pod \"image-registry-d78b99cdd-bhxqb\" (UID: \"d297493c-df72-413e-988f-c4d38b6bbf5f\") " pod="openshift-image-registry/image-registry-d78b99cdd-bhxqb"
Sep 09 15:24:37.518996 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:37.518975    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d297493c-df72-413e-988f-c4d38b6bbf5f-bound-sa-token\") pod \"image-registry-d78b99cdd-bhxqb\" (UID: \"d297493c-df72-413e-988f-c4d38b6bbf5f\") " pod="openshift-image-registry/image-registry-d78b99cdd-bhxqb"
Sep 09 15:24:37.638661 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:37.638570    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d78b99cdd-bhxqb"
Sep 09 15:24:37.909482 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:37.909409    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d78b99cdd-bhxqb"]
Sep 09 15:24:37.911870 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:24:37.911844    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd297493c_df72_413e_988f_c4d38b6bbf5f.slice/crio-137dc75a232a12283e948a27d3c24414d12c0e964a1a933dc3c821711fad8c75 WatchSource:0}: Error finding container 137dc75a232a12283e948a27d3c24414d12c0e964a1a933dc3c821711fad8c75: Status 404 returned error can't find the container with id 137dc75a232a12283e948a27d3c24414d12c0e964a1a933dc3c821711fad8c75
Sep 09 15:24:38.730350 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:38.730317    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d78b99cdd-bhxqb" event={"ID":"d297493c-df72-413e-988f-c4d38b6bbf5f","Type":"ContainerStarted","Data":"c75fb4e85ca313f05eb4a613d79c5c3c14a06bb61bcd10f26ef569ac5f1bf3f2"}
Sep 09 15:24:38.730782 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:38.730371    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d78b99cdd-bhxqb" event={"ID":"d297493c-df72-413e-988f-c4d38b6bbf5f","Type":"ContainerStarted","Data":"137dc75a232a12283e948a27d3c24414d12c0e964a1a933dc3c821711fad8c75"}
Sep 09 15:24:38.730782 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:38.730430    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-d78b99cdd-bhxqb"
Sep 09 15:24:38.749362 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:38.749313    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-d78b99cdd-bhxqb" podStartSLOduration=1.749298217 podStartE2EDuration="1.749298217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:24:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:24:38.748401027 +0000 UTC m=+244.347295671" watchObservedRunningTime="2026-09-09 15:24:38.749298217 +0000 UTC m=+244.348192860"
Sep 09 15:24:46.775752 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:46.775695    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57473bc0-f6d0-45f1-8954-93a7d2f4321b-metrics-certs\") pod \"network-metrics-daemon-v5kbj\" (UID: \"57473bc0-f6d0-45f1-8954-93a7d2f4321b\") " pod="openshift-multus/network-metrics-daemon-v5kbj"
Sep 09 15:24:46.777978 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:46.777957    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57473bc0-f6d0-45f1-8954-93a7d2f4321b-metrics-certs\") pod \"network-metrics-daemon-v5kbj\" (UID: \"57473bc0-f6d0-45f1-8954-93a7d2f4321b\") " pod="openshift-multus/network-metrics-daemon-v5kbj"
Sep 09 15:24:46.959037 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:46.959010    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xdr6q\""
Sep 09 15:24:46.967211 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:46.967189    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5kbj"
Sep 09 15:24:47.233761 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:47.233714    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v5kbj"]
Sep 09 15:24:47.236756 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:24:47.236713    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57473bc0_f6d0_45f1_8954_93a7d2f4321b.slice/crio-292a29d1398bc05cf7f4d80a5b9fe72f76a5ccb0a2d7f57ed0fab867e2291b61 WatchSource:0}: Error finding container 292a29d1398bc05cf7f4d80a5b9fe72f76a5ccb0a2d7f57ed0fab867e2291b61: Status 404 returned error can't find the container with id 292a29d1398bc05cf7f4d80a5b9fe72f76a5ccb0a2d7f57ed0fab867e2291b61
Sep 09 15:24:47.480211 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:47.480167    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hz29k\" (UID: \"f3f904f8-3a34-41d2-8f0c-0695ad8b5209\") " pod="openshift-insights/insights-runtime-extractor-hz29k"
Sep 09 15:24:47.482822 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:47.482798    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3f904f8-3a34-41d2-8f0c-0695ad8b5209-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hz29k\" (UID: \"f3f904f8-3a34-41d2-8f0c-0695ad8b5209\") " pod="openshift-insights/insights-runtime-extractor-hz29k"
Sep 09 15:24:47.751490 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:47.751461    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6tbnq\""
Sep 09 15:24:47.756940 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:47.756910    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v5kbj" event={"ID":"57473bc0-f6d0-45f1-8954-93a7d2f4321b","Type":"ContainerStarted","Data":"292a29d1398bc05cf7f4d80a5b9fe72f76a5ccb0a2d7f57ed0fab867e2291b61"}
Sep 09 15:24:47.759072 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:47.759057    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hz29k"
Sep 09 15:24:48.026331 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:48.026068    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hz29k"]
Sep 09 15:24:48.029130 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:24:48.029096    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f904f8_3a34_41d2_8f0c_0695ad8b5209.slice/crio-86ffca42ec59029f0301cad97127f8c2fad6372b7c697cdd3976698d5a001ec3 WatchSource:0}: Error finding container 86ffca42ec59029f0301cad97127f8c2fad6372b7c697cdd3976698d5a001ec3: Status 404 returned error can't find the container with id 86ffca42ec59029f0301cad97127f8c2fad6372b7c697cdd3976698d5a001ec3
Sep 09 15:24:48.762096 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:48.762008    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v5kbj" event={"ID":"57473bc0-f6d0-45f1-8954-93a7d2f4321b","Type":"ContainerStarted","Data":"91c0c1e02ad8091c823eb3a367fc3753d8fc7c073c2b35259be1cbfd3146fa99"}
Sep 09 15:24:48.762096 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:48.762066    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v5kbj" event={"ID":"57473bc0-f6d0-45f1-8954-93a7d2f4321b","Type":"ContainerStarted","Data":"0d4ad9bb03684e84ed1be41a4c411b210105e43b4bc6e248ff4c03d329bf4395"}
Sep 09 15:24:48.763516 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:48.763496    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hz29k" event={"ID":"f3f904f8-3a34-41d2-8f0c-0695ad8b5209","Type":"ContainerStarted","Data":"8d5e202f66f506271430b19d31e0d12524cef180a88f0fbbcced0706cf1f03a9"}
Sep 09 15:24:48.763581 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:48.763539    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hz29k" event={"ID":"f3f904f8-3a34-41d2-8f0c-0695ad8b5209","Type":"ContainerStarted","Data":"86ffca42ec59029f0301cad97127f8c2fad6372b7c697cdd3976698d5a001ec3"}
Sep 09 15:24:48.776153 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:48.776106    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-v5kbj" podStartSLOduration=252.50832469 podStartE2EDuration="4m13.776092342s" totalImagesPullingTime="1.267767652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:20:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:24:48.775676047 +0000 UTC m=+254.374570692" watchObservedRunningTime="2026-09-09 15:24:48.776092342 +0000 UTC m=+254.374986986"
Sep 09 15:24:50.771693 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:50.771655    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hz29k" event={"ID":"f3f904f8-3a34-41d2-8f0c-0695ad8b5209","Type":"ContainerStarted","Data":"d16c4e8fc077c992b596c88ab1ea6e1ca996118e661197e81e8d732cff2980fc"}
Sep 09 15:24:52.557088 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:52.557055    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-n9qh8"]
Sep 09 15:24:52.560075 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:52.560056    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n9qh8"
Sep 09 15:24:52.563150 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:52.563113    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 15:24:52.563275 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:52.563184    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 15:24:52.563275 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:52.563237    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 15:24:52.564510 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:52.564292    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7vtq5\""
Sep 09 15:24:52.564510 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:52.564343    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 15:24:52.564510 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:52.564352    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 15:24:52.564510 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:52.564395    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 15:24:52.623399 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:52.623352    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6654b5d1-f133-49eb-9062-74548d0e0670-node-exporter-textfile\") pod \"node-exporter-n9qh8\" (UID: \"6654b5d1-f133-49eb-9062-74548d0e0670\") " pod="openshift-monitoring/node-exporter-n9qh8"
Sep 09 15:24:52.623399 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:52.623400    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6654b5d1-f133-49eb-9062-74548d0e0670-node-exporter-wtmp\") pod \"node-exporter-n9qh8\" (UID: \"6654b5d1-f133-49eb-9062-74548d0e0670\") " pod="openshift-monitoring/node-exporter-n9qh8"
Sep 09 15:24:52.623649 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:52.623434    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6654b5d1-f133-49eb-9062-74548d0e0670-node-exporter-tls\") pod \"node-exporter-n9qh8\" (UID: \"6654b5d1-f133-49eb-9062-74548d0e0670\") " pod="openshift-monitoring/node-exporter-n9qh8"
Sep 09 15:24:52.623649 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:52.623495    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz976\" (UniqueName: \"kubernetes.io/projected/6654b5d1-f133-49eb-9062-74548d0e0670-kube-api-access-bz976\") pod \"node-exporter-n9qh8\" (UID: \"6654b5d1-f133-49eb-9062-74548d0e0670\") " pod="openshift-monitoring/node-exporter-n9qh8"
Sep 09 15:24:52.623649 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:52.623556    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6654b5d1-f133-49eb-9062-74548d0e0670-metrics-client-ca\") pod \"node-exporter-n9qh8\" (UID: \"6654b5d1-f133-49eb-9062-74548d0e0670\") " pod="openshift-monitoring/node-exporter-n9qh8"
Sep 09 15:24:52.623649 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:52.623584    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6654b5d1-f133-49eb-9062-74548d0e0670-sys\") pod \"node-exporter-n9qh8\" (UID: \"6654b5d1-f133-49eb-9062-74548d0e0670\") " pod="openshift-monitoring/node-exporter-n9qh8"
Sep 09 15:24:52.623649 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:52.623623    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6654b5d1-f133-49eb-9062-74548d0e0670-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n9qh8\" (UID: \"6654b5d1-f133-49eb-9062-74548d0e0670\") " pod="openshift-monitoring/node-exporter-n9qh8"
Sep 09 15:24:52.623919 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:52.623653    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6654b5d1-f133-49eb-9062-74548d0e0670-root\") pod \"node-exporter-n9qh8\" (UID: \"6654b5d1-f133-49eb-9062-74548d0e0670\") " pod="openshift-monitoring/node-exporter-n9qh8"
Sep 09 15:24:52.623919 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:52.623716    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6654b5d1-f133-49eb-9062-74548d0e0670-node-exporter-accelerators-collector-config\") pod \"node-exporter-n9qh8\" (UID: \"6654b5d1-f133-49eb-9062-74548d0e0670\") " pod="openshift-monitoring/node-exporter-n9qh8"
Sep 09 15:24:52.724880 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:52.724827    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6654b5d1-f133-49eb-9062-74548d0e0670-node-exporter-accelerators-collector-config\") pod \"node-exporter-n9qh8\" (UID: \"6654b5d1-f133-49eb-9062-74548d0e0670\") " pod="openshift-monitoring/node-exporter-n9qh8"
Sep 09 15:24:52.724880 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:52.724889    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6654b5d1-f133-49eb-9062-74548d0e0670-node-exporter-textfile\") pod \"node-exporter-n9qh8\" (UID: \"6654b5d1-f133-49eb-9062-74548d0e0670\") " pod="openshift-monitoring/node-exporter-n9qh8"
Sep 09 15:24:52.725140 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:52.724977    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6654b5d1-f133-49eb-9062-74548d0e0670-node-exporter-wtmp\") pod \"node-exporter-n9qh8\" (UID: \"6654b5d1-f133-49eb-9062-74548d0e0670\") " pod="openshift-monitoring/node-exporter-n9qh8"
Sep 09 15:24:52.725140 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:52.725009    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6654b5d1-f133-49eb-9062-74548d0e0670-node-exporter-tls\") pod \"node-exporter-n9qh8\" (UID: \"6654b5d1-f133-49eb-9062-74548d0e0670\") " pod="openshift-monitoring/node-exporter-n9qh8"
Sep 09 15:24:52.725140 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:52.725029    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz976\" (UniqueName: \"kubernetes.io/projected/6654b5d1-f133-49eb-9062-74548d0e0670-kube-api-access-bz976\") pod \"node-exporter-n9qh8\" (UID: \"6654b5d1-f133-49eb-9062-74548d0e0670\") " pod="openshift-monitoring/node-exporter-n9qh8"
Sep 09 15:24:52.725140 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:52.725062    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6654b5d1-f133-49eb-9062-74548d0e0670-metrics-client-ca\") pod \"node-exporter-n9qh8\" (UID: \"6654b5d1-f133-49eb-9062-74548d0e0670\") " pod="openshift-monitoring/node-exporter-n9qh8"
Sep 09 15:24:52.725140 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:52.725085    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6654b5d1-f133-49eb-9062-74548d0e0670-sys\") pod \"node-exporter-n9qh8\" (UID: \"6654b5d1-f133-49eb-9062-74548d0e0670\") " pod="openshift-monitoring/node-exporter-n9qh8"
Sep 09 15:24:52.725140 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:52.725114    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6654b5d1-f133-49eb-9062-74548d0e0670-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n9qh8\" (UID: \"6654b5d1-f133-49eb-9062-74548d0e0670\") " pod="openshift-monitoring/node-exporter-n9qh8"
Sep 09 15:24:52.725140 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:52.725138    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6654b5d1-f133-49eb-9062-74548d0e0670-root\") pod \"node-exporter-n9qh8\" (UID: \"6654b5d1-f133-49eb-9062-74548d0e0670\") " pod="openshift-monitoring/node-exporter-n9qh8"
Sep 09 15:24:52.725493 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:24:52.725180    2553 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 15:24:52.725493 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:52.725214    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6654b5d1-f133-49eb-9062-74548d0e0670-root\") pod \"node-exporter-n9qh8\" (UID: \"6654b5d1-f133-49eb-9062-74548d0e0670\") " pod="openshift-monitoring/node-exporter-n9qh8"
Sep 09 15:24:52.725493 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:24:52.725270    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6654b5d1-f133-49eb-9062-74548d0e0670-node-exporter-tls podName:6654b5d1-f133-49eb-9062-74548d0e0670 nodeName:}" failed. No retries permitted until 2026-09-09 15:24:53.225249144 +0000 UTC m=+258.824143770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/6654b5d1-f133-49eb-9062-74548d0e0670-node-exporter-tls") pod "node-exporter-n9qh8" (UID: "6654b5d1-f133-49eb-9062-74548d0e0670") : secret "node-exporter-tls" not found
Sep 09 15:24:52.725493 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:52.725319    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6654b5d1-f133-49eb-9062-74548d0e0670-node-exporter-textfile\") pod \"node-exporter-n9qh8\" (UID: \"6654b5d1-f133-49eb-9062-74548d0e0670\") " pod="openshift-monitoring/node-exporter-n9qh8"
Sep 09 15:24:52.725493 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:52.725393    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6654b5d1-f133-49eb-9062-74548d0e0670-sys\") pod \"node-exporter-n9qh8\" (UID: \"6654b5d1-f133-49eb-9062-74548d0e0670\") " pod="openshift-monitoring/node-exporter-n9qh8"
Sep 09 15:24:52.725493 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:52.725472    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6654b5d1-f133-49eb-9062-74548d0e0670-node-exporter-wtmp\") pod \"node-exporter-n9qh8\" (UID: \"6654b5d1-f133-49eb-9062-74548d0e0670\") " pod="openshift-monitoring/node-exporter-n9qh8"
Sep 09 15:24:52.725835 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:52.725668    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6654b5d1-f133-49eb-9062-74548d0e0670-metrics-client-ca\") pod \"node-exporter-n9qh8\" (UID: \"6654b5d1-f133-49eb-9062-74548d0e0670\") " pod="openshift-monitoring/node-exporter-n9qh8"
Sep 09 15:24:52.726131 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:52.726109    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6654b5d1-f133-49eb-9062-74548d0e0670-node-exporter-accelerators-collector-config\") pod \"node-exporter-n9qh8\" (UID: \"6654b5d1-f133-49eb-9062-74548d0e0670\") " pod="openshift-monitoring/node-exporter-n9qh8"
Sep 09 15:24:52.727549 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:52.727532    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6654b5d1-f133-49eb-9062-74548d0e0670-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n9qh8\" (UID: \"6654b5d1-f133-49eb-9062-74548d0e0670\") " pod="openshift-monitoring/node-exporter-n9qh8"
Sep 09 15:24:52.735434 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:52.735409    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz976\" (UniqueName: \"kubernetes.io/projected/6654b5d1-f133-49eb-9062-74548d0e0670-kube-api-access-bz976\") pod \"node-exporter-n9qh8\" (UID: \"6654b5d1-f133-49eb-9062-74548d0e0670\") " pod="openshift-monitoring/node-exporter-n9qh8"
Sep 09 15:24:52.785754 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:52.785643    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hz29k" event={"ID":"f3f904f8-3a34-41d2-8f0c-0695ad8b5209","Type":"ContainerStarted","Data":"b34805ba035f22e8c7fd89e7d2cd9dffe8f9377579229cd5fb83871ef2ffbd03"}
Sep 09 15:24:52.804950 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:52.804895    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hz29k" podStartSLOduration=34.061426087 podStartE2EDuration="37.804878335s" totalImagesPullingTime="3.743452248s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:24:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:24:52.804585008 +0000 UTC m=+258.403479653" watchObservedRunningTime="2026-09-09 15:24:52.804878335 +0000 UTC m=+258.403772978"
Sep 09 15:24:53.230362 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:53.230316    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6654b5d1-f133-49eb-9062-74548d0e0670-node-exporter-tls\") pod \"node-exporter-n9qh8\" (UID: \"6654b5d1-f133-49eb-9062-74548d0e0670\") " pod="openshift-monitoring/node-exporter-n9qh8"
Sep 09 15:24:53.232597 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:53.232565    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6654b5d1-f133-49eb-9062-74548d0e0670-node-exporter-tls\") pod \"node-exporter-n9qh8\" (UID: \"6654b5d1-f133-49eb-9062-74548d0e0670\") " pod="openshift-monitoring/node-exporter-n9qh8"
Sep 09 15:24:53.470538 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:53.470501    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n9qh8"
Sep 09 15:24:53.484552 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:24:53.482232    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6654b5d1_f133_49eb_9062_74548d0e0670.slice/crio-899ce0e8c47e2416f57a2c8d19bbbf98cc97b9566fa50610cf65b7df13b7354c WatchSource:0}: Error finding container 899ce0e8c47e2416f57a2c8d19bbbf98cc97b9566fa50610cf65b7df13b7354c: Status 404 returned error can't find the container with id 899ce0e8c47e2416f57a2c8d19bbbf98cc97b9566fa50610cf65b7df13b7354c
Sep 09 15:24:53.790003 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:53.789902    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n9qh8" event={"ID":"6654b5d1-f133-49eb-9062-74548d0e0670","Type":"ContainerStarted","Data":"899ce0e8c47e2416f57a2c8d19bbbf98cc97b9566fa50610cf65b7df13b7354c"}
Sep 09 15:24:55.798906 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:55.798872    2553 generic.go:356] "Generic (PLEG): container finished" podID="6654b5d1-f133-49eb-9062-74548d0e0670" containerID="f71634894b0bb1a824edfc30810cbb3080bc79a21faf50cf81e687f5eaed05f4" exitCode=0
Sep 09 15:24:55.799376 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:55.798927    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n9qh8" event={"ID":"6654b5d1-f133-49eb-9062-74548d0e0670","Type":"ContainerDied","Data":"f71634894b0bb1a824edfc30810cbb3080bc79a21faf50cf81e687f5eaed05f4"}
Sep 09 15:24:56.805158 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:56.805120    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n9qh8" event={"ID":"6654b5d1-f133-49eb-9062-74548d0e0670","Type":"ContainerStarted","Data":"528428c087754e81577c200d425cfb6d0233cb062b16d24daaa7af7376ce07f5"}
Sep 09 15:24:56.805542 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:56.805188    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n9qh8" event={"ID":"6654b5d1-f133-49eb-9062-74548d0e0670","Type":"ContainerStarted","Data":"babddd370a831762c3f689a645239968c3fd3fa20cf973cd07c54673769a54d0"}
Sep 09 15:24:56.827558 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:56.827511    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-n9qh8" podStartSLOduration=3.143190376 podStartE2EDuration="4.827494176s" totalImagesPullingTime="1.6237887s" totalInitContainerRuntime="60.5151ms" isStatefulPod=true podCreationTimestamp="2026-09-09 15:24:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:24:56.826503165 +0000 UTC m=+262.425397809" watchObservedRunningTime="2026-09-09 15:24:56.827494176 +0000 UTC m=+262.426388820"
Sep 09 15:24:57.645485 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:57.645441    2553 patch_prober.go:28] interesting pod/image-registry-d78b99cdd-bhxqb container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 15:24:57.645676 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:57.645506    2553 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-d78b99cdd-bhxqb" podUID="d297493c-df72-413e-988f-c4d38b6bbf5f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 15:24:59.737350 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:59.737307    2553 patch_prober.go:28] interesting pod/image-registry-d78b99cdd-bhxqb container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 15:24:59.737727 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:24:59.737362    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-d78b99cdd-bhxqb" podUID="d297493c-df72-413e-988f-c4d38b6bbf5f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 15:25:07.643595 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:25:07.643563    2553 patch_prober.go:28] interesting pod/image-registry-d78b99cdd-bhxqb container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 15:25:07.644001 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:25:07.643614    2553 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-d78b99cdd-bhxqb" podUID="d297493c-df72-413e-988f-c4d38b6bbf5f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 15:25:09.736994 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:25:09.736966    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-d78b99cdd-bhxqb"
Sep 09 15:25:15.468551 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:25:15.468511    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-cwkld" podUID="acc8de36-2e42-4471-a31e-38c66f87ab8c"
Sep 09 15:25:15.863047 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:25:15.862969    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cwkld"
Sep 09 15:25:19.134527 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:25:19.134488    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acc8de36-2e42-4471-a31e-38c66f87ab8c-metrics-tls\") pod \"dns-default-cwkld\" (UID: \"acc8de36-2e42-4471-a31e-38c66f87ab8c\") " pod="openshift-dns/dns-default-cwkld"
Sep 09 15:25:19.136811 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:25:19.136788    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acc8de36-2e42-4471-a31e-38c66f87ab8c-metrics-tls\") pod \"dns-default-cwkld\" (UID: \"acc8de36-2e42-4471-a31e-38c66f87ab8c\") " pod="openshift-dns/dns-default-cwkld"
Sep 09 15:25:19.166512 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:25:19.166479    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jnqpd\""
Sep 09 15:25:19.174704 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:25:19.174682    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cwkld"
Sep 09 15:25:19.451033 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:25:19.451002    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cwkld"]
Sep 09 15:25:19.453848 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:25:19.453818    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc8de36_2e42_4471_a31e_38c66f87ab8c.slice/crio-d5894078f9aed2f5bebf0004840221b4e2c80bd97e821b84b991b5ca603fbc74 WatchSource:0}: Error finding container d5894078f9aed2f5bebf0004840221b4e2c80bd97e821b84b991b5ca603fbc74: Status 404 returned error can't find the container with id d5894078f9aed2f5bebf0004840221b4e2c80bd97e821b84b991b5ca603fbc74
Sep 09 15:25:19.879265 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:25:19.879186    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cwkld" event={"ID":"acc8de36-2e42-4471-a31e-38c66f87ab8c","Type":"ContainerStarted","Data":"d5894078f9aed2f5bebf0004840221b4e2c80bd97e821b84b991b5ca603fbc74"}
Sep 09 15:25:21.219237 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:25:21.219144    2553 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6967d4f-t7dwt" podUID="8ed49a29-af63-4d51-8729-eb4fcc081daa" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 09 15:25:21.888099 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:25:21.888064    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cwkld" event={"ID":"acc8de36-2e42-4471-a31e-38c66f87ab8c","Type":"ContainerStarted","Data":"cf2f6c01fdc4820695f4656d745898c0b5800b1c149a64e196e28cbcc21fd069"}
Sep 09 15:25:21.888263 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:25:21.888147    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cwkld" event={"ID":"acc8de36-2e42-4471-a31e-38c66f87ab8c","Type":"ContainerStarted","Data":"b0e4c8fe3bea0133f496a6efd11551e27fd545f9ecaec4d5deda53fc942d18aa"}
Sep 09 15:25:21.888263 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:25:21.888198    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-cwkld"
Sep 09 15:25:21.906961 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:25:21.906915    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-cwkld" podStartSLOduration=251.430124479 podStartE2EDuration="4m12.906900554s" totalImagesPullingTime="1.476776075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:21:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:25:21.906508176 +0000 UTC m=+287.505402835" watchObservedRunningTime="2026-09-09 15:25:21.906900554 +0000 UTC m=+287.505795197"
Sep 09 15:25:31.218960 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:25:31.218923    2553 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6967d4f-t7dwt" podUID="8ed49a29-af63-4d51-8729-eb4fcc081daa" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 09 15:25:31.893785 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:25:31.893754    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-cwkld"
Sep 09 15:25:34.789978 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:25:34.789953    2553 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 15:25:41.218641 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:25:41.218604    2553 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6967d4f-t7dwt" podUID="8ed49a29-af63-4d51-8729-eb4fcc081daa" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 09 15:25:41.220952 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:25:41.218680    2553 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6967d4f-t7dwt"
Sep 09 15:25:41.220952 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:25:41.219167    2553 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"fa757bdfa297052ac6c540509b07401d1a41046fd3a04d3e90e00a060b429f6e"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6967d4f-t7dwt" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 09 15:25:41.220952 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:25:41.219215    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6967d4f-t7dwt" podUID="8ed49a29-af63-4d51-8729-eb4fcc081daa" containerName="service-proxy" containerID="cri-o://fa757bdfa297052ac6c540509b07401d1a41046fd3a04d3e90e00a060b429f6e" gracePeriod=30
Sep 09 15:25:41.954523 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:25:41.954488    2553 generic.go:356] "Generic (PLEG): container finished" podID="8ed49a29-af63-4d51-8729-eb4fcc081daa" containerID="fa757bdfa297052ac6c540509b07401d1a41046fd3a04d3e90e00a060b429f6e" exitCode=2
Sep 09 15:25:41.954703 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:25:41.954565    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6967d4f-t7dwt" event={"ID":"8ed49a29-af63-4d51-8729-eb4fcc081daa","Type":"ContainerDied","Data":"fa757bdfa297052ac6c540509b07401d1a41046fd3a04d3e90e00a060b429f6e"}
Sep 09 15:25:41.954703 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:25:41.954613    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6967d4f-t7dwt" event={"ID":"8ed49a29-af63-4d51-8729-eb4fcc081daa","Type":"ContainerStarted","Data":"541c2b3d0a2bdb59e77435f4c451383fe7a9d5ad7d4cb9d931a95ed015669f71"}
Sep 09 15:29:36.751149 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:36.751112    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mnnvk"]
Sep 09 15:29:36.754065 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:36.754048    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mnnvk"
Sep 09 15:29:36.756709 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:36.756689    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 09 15:29:36.756828 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:36.756723    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-bvj54\""
Sep 09 15:29:36.757706 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:36.757688    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 09 15:29:36.765037 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:36.765010    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mnnvk"]
Sep 09 15:29:36.841302 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:36.841261    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b81841cc-d69c-4568-99af-252d0c1f8996-tmp\") pod \"openshift-lws-operator-fd8ccff4c-mnnvk\" (UID: \"b81841cc-d69c-4568-99af-252d0c1f8996\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mnnvk"
Sep 09 15:29:36.841302 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:36.841300    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh5z6\" (UniqueName: \"kubernetes.io/projected/b81841cc-d69c-4568-99af-252d0c1f8996-kube-api-access-nh5z6\") pod \"openshift-lws-operator-fd8ccff4c-mnnvk\" (UID: \"b81841cc-d69c-4568-99af-252d0c1f8996\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mnnvk"
Sep 09 15:29:36.942304 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:36.942270    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b81841cc-d69c-4568-99af-252d0c1f8996-tmp\") pod \"openshift-lws-operator-fd8ccff4c-mnnvk\" (UID: \"b81841cc-d69c-4568-99af-252d0c1f8996\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mnnvk"
Sep 09 15:29:36.942418 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:36.942312    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh5z6\" (UniqueName: \"kubernetes.io/projected/b81841cc-d69c-4568-99af-252d0c1f8996-kube-api-access-nh5z6\") pod \"openshift-lws-operator-fd8ccff4c-mnnvk\" (UID: \"b81841cc-d69c-4568-99af-252d0c1f8996\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mnnvk"
Sep 09 15:29:36.942699 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:36.942674    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b81841cc-d69c-4568-99af-252d0c1f8996-tmp\") pod \"openshift-lws-operator-fd8ccff4c-mnnvk\" (UID: \"b81841cc-d69c-4568-99af-252d0c1f8996\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mnnvk"
Sep 09 15:29:36.950769 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:36.950725    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh5z6\" (UniqueName: \"kubernetes.io/projected/b81841cc-d69c-4568-99af-252d0c1f8996-kube-api-access-nh5z6\") pod \"openshift-lws-operator-fd8ccff4c-mnnvk\" (UID: \"b81841cc-d69c-4568-99af-252d0c1f8996\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mnnvk"
Sep 09 15:29:37.062541 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:37.062443    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mnnvk"
Sep 09 15:29:37.334290 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:37.334208    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mnnvk"]
Sep 09 15:29:37.337348 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:29:37.337309    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb81841cc_d69c_4568_99af_252d0c1f8996.slice/crio-edd165b86f2c92e83db85db01a5aa6158aedd735b3b25d7759316a8f455f6cb7 WatchSource:0}: Error finding container edd165b86f2c92e83db85db01a5aa6158aedd735b3b25d7759316a8f455f6cb7: Status 404 returned error can't find the container with id edd165b86f2c92e83db85db01a5aa6158aedd735b3b25d7759316a8f455f6cb7
Sep 09 15:29:37.338821 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:37.338799    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 15:29:37.724564 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:37.724528    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mnnvk" event={"ID":"b81841cc-d69c-4568-99af-252d0c1f8996","Type":"ContainerStarted","Data":"edd165b86f2c92e83db85db01a5aa6158aedd735b3b25d7759316a8f455f6cb7"}
Sep 09 15:29:40.738196 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:40.738161    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mnnvk" event={"ID":"b81841cc-d69c-4568-99af-252d0c1f8996","Type":"ContainerStarted","Data":"bcb5be26c1bd8d9078a5664fc5ea1b91bb9e84f5cc7813e8b09ba762c37335cc"}
Sep 09 15:29:40.760929 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:40.760875    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mnnvk" podStartSLOduration=2.17925514 podStartE2EDuration="4.760858587s" totalImagesPullingTime="2.581603447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:29:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:29:40.760078648 +0000 UTC m=+546.358973294" watchObservedRunningTime="2026-09-09 15:29:40.760858587 +0000 UTC m=+546.359753245"
Sep 09 15:29:55.404302 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:55.404264    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-76f449f6db-2j5kb"]
Sep 09 15:29:55.407346 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:55.407328    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-76f449f6db-2j5kb"
Sep 09 15:29:55.409924 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:55.409894    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 09 15:29:55.410067 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:55.409900    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 09 15:29:55.410067 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:55.410007    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-r757k\""
Sep 09 15:29:55.410165 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:55.410088    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 09 15:29:55.416184 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:55.416157    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-76f449f6db-2j5kb"]
Sep 09 15:29:55.472461 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:55.472412    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4nnm\" (UniqueName: \"kubernetes.io/projected/597e9f25-fee1-4d02-b59f-695e293df055-kube-api-access-l4nnm\") pod \"lws-controller-manager-76f449f6db-2j5kb\" (UID: \"597e9f25-fee1-4d02-b59f-695e293df055\") " pod="openshift-lws-operator/lws-controller-manager-76f449f6db-2j5kb"
Sep 09 15:29:55.472621 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:55.472514    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/597e9f25-fee1-4d02-b59f-695e293df055-metrics-cert\") pod \"lws-controller-manager-76f449f6db-2j5kb\" (UID: \"597e9f25-fee1-4d02-b59f-695e293df055\") " pod="openshift-lws-operator/lws-controller-manager-76f449f6db-2j5kb"
Sep 09 15:29:55.472621 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:55.472547    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/597e9f25-fee1-4d02-b59f-695e293df055-manager-config\") pod \"lws-controller-manager-76f449f6db-2j5kb\" (UID: \"597e9f25-fee1-4d02-b59f-695e293df055\") " pod="openshift-lws-operator/lws-controller-manager-76f449f6db-2j5kb"
Sep 09 15:29:55.472621 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:55.472612    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/597e9f25-fee1-4d02-b59f-695e293df055-cert\") pod \"lws-controller-manager-76f449f6db-2j5kb\" (UID: \"597e9f25-fee1-4d02-b59f-695e293df055\") " pod="openshift-lws-operator/lws-controller-manager-76f449f6db-2j5kb"
Sep 09 15:29:55.573275 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:55.573240    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/597e9f25-fee1-4d02-b59f-695e293df055-metrics-cert\") pod \"lws-controller-manager-76f449f6db-2j5kb\" (UID: \"597e9f25-fee1-4d02-b59f-695e293df055\") " pod="openshift-lws-operator/lws-controller-manager-76f449f6db-2j5kb"
Sep 09 15:29:55.573275 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:55.573276    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/597e9f25-fee1-4d02-b59f-695e293df055-manager-config\") pod \"lws-controller-manager-76f449f6db-2j5kb\" (UID: \"597e9f25-fee1-4d02-b59f-695e293df055\") " pod="openshift-lws-operator/lws-controller-manager-76f449f6db-2j5kb"
Sep 09 15:29:55.573482 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:55.573310    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/597e9f25-fee1-4d02-b59f-695e293df055-cert\") pod \"lws-controller-manager-76f449f6db-2j5kb\" (UID: \"597e9f25-fee1-4d02-b59f-695e293df055\") " pod="openshift-lws-operator/lws-controller-manager-76f449f6db-2j5kb"
Sep 09 15:29:55.573482 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:55.573352    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4nnm\" (UniqueName: \"kubernetes.io/projected/597e9f25-fee1-4d02-b59f-695e293df055-kube-api-access-l4nnm\") pod \"lws-controller-manager-76f449f6db-2j5kb\" (UID: \"597e9f25-fee1-4d02-b59f-695e293df055\") " pod="openshift-lws-operator/lws-controller-manager-76f449f6db-2j5kb"
Sep 09 15:29:55.574034 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:55.574013    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/597e9f25-fee1-4d02-b59f-695e293df055-manager-config\") pod \"lws-controller-manager-76f449f6db-2j5kb\" (UID: \"597e9f25-fee1-4d02-b59f-695e293df055\") " pod="openshift-lws-operator/lws-controller-manager-76f449f6db-2j5kb"
Sep 09 15:29:55.575921 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:55.575899    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/597e9f25-fee1-4d02-b59f-695e293df055-cert\") pod \"lws-controller-manager-76f449f6db-2j5kb\" (UID: \"597e9f25-fee1-4d02-b59f-695e293df055\") " pod="openshift-lws-operator/lws-controller-manager-76f449f6db-2j5kb"
Sep 09 15:29:55.576001 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:55.575945    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/597e9f25-fee1-4d02-b59f-695e293df055-metrics-cert\") pod \"lws-controller-manager-76f449f6db-2j5kb\" (UID: \"597e9f25-fee1-4d02-b59f-695e293df055\") " pod="openshift-lws-operator/lws-controller-manager-76f449f6db-2j5kb"
Sep 09 15:29:55.583240 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:55.583210    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4nnm\" (UniqueName: \"kubernetes.io/projected/597e9f25-fee1-4d02-b59f-695e293df055-kube-api-access-l4nnm\") pod \"lws-controller-manager-76f449f6db-2j5kb\" (UID: \"597e9f25-fee1-4d02-b59f-695e293df055\") " pod="openshift-lws-operator/lws-controller-manager-76f449f6db-2j5kb"
Sep 09 15:29:55.717167 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:55.717118    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-76f449f6db-2j5kb"
Sep 09 15:29:55.996061 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:55.995971    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-76f449f6db-2j5kb"]
Sep 09 15:29:55.999479 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:29:55.999448    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod597e9f25_fee1_4d02_b59f_695e293df055.slice/crio-82736c83f97a606031ab1afa7b4e1ff582f035c0b581be0402ad1854faa0ea55 WatchSource:0}: Error finding container 82736c83f97a606031ab1afa7b4e1ff582f035c0b581be0402ad1854faa0ea55: Status 404 returned error can't find the container with id 82736c83f97a606031ab1afa7b4e1ff582f035c0b581be0402ad1854faa0ea55
Sep 09 15:29:56.794967 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:56.794932    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-76f449f6db-2j5kb" event={"ID":"597e9f25-fee1-4d02-b59f-695e293df055","Type":"ContainerStarted","Data":"82736c83f97a606031ab1afa7b4e1ff582f035c0b581be0402ad1854faa0ea55"}
Sep 09 15:29:58.804042 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:58.804007    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-76f449f6db-2j5kb" event={"ID":"597e9f25-fee1-4d02-b59f-695e293df055","Type":"ContainerStarted","Data":"2536fb4873aabba0a6d44383125696ae510d0f186486641ee27915fba2be26ff"}
Sep 09 15:29:58.804449 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:58.804103    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-76f449f6db-2j5kb"
Sep 09 15:29:58.821523 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:29:58.821468    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-76f449f6db-2j5kb" podStartSLOduration=1.965757606 podStartE2EDuration="3.821453954s" totalImagesPullingTime="1.855696348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:29:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:29:58.82081954 +0000 UTC m=+564.419714185" watchObservedRunningTime="2026-09-09 15:29:58.821453954 +0000 UTC m=+564.420348720"
Sep 09 15:30:00.167132 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:00.167091    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74fc54996-kqp5k"]
Sep 09 15:30:00.170485 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:00.170461    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-kqp5k"
Sep 09 15:30:00.175011 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:00.174987    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 09 15:30:00.175011 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:00.175005    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 09 15:30:00.175212 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:00.175126    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 09 15:30:00.175292 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:00.175279    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-hpnwk\""
Sep 09 15:30:00.175564 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:00.175551    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 09 15:30:00.204946 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:00.204912    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74fc54996-kqp5k"]
Sep 09 15:30:00.311495 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:00.311447    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fb5de8f7-ab6e-426e-9f58-e57de2f6acac-manifests\") pod \"opendatahub-operator-controller-manager-74fc54996-kqp5k\" (UID: \"fb5de8f7-ab6e-426e-9f58-e57de2f6acac\") " pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-kqp5k"
Sep 09 15:30:00.311495 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:00.311491    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb5de8f7-ab6e-426e-9f58-e57de2f6acac-tmp\") pod \"opendatahub-operator-controller-manager-74fc54996-kqp5k\" (UID: \"fb5de8f7-ab6e-426e-9f58-e57de2f6acac\") " pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-kqp5k"
Sep 09 15:30:00.311707 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:00.311522    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qfm5\" (UniqueName: \"kubernetes.io/projected/fb5de8f7-ab6e-426e-9f58-e57de2f6acac-kube-api-access-2qfm5\") pod \"opendatahub-operator-controller-manager-74fc54996-kqp5k\" (UID: \"fb5de8f7-ab6e-426e-9f58-e57de2f6acac\") " pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-kqp5k"
Sep 09 15:30:00.311707 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:00.311555    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fb5de8f7-ab6e-426e-9f58-e57de2f6acac-webhook-cert\") pod \"opendatahub-operator-controller-manager-74fc54996-kqp5k\" (UID: \"fb5de8f7-ab6e-426e-9f58-e57de2f6acac\") " pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-kqp5k"
Sep 09 15:30:00.311707 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:00.311611    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fb5de8f7-ab6e-426e-9f58-e57de2f6acac-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74fc54996-kqp5k\" (UID: \"fb5de8f7-ab6e-426e-9f58-e57de2f6acac\") " pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-kqp5k"
Sep 09 15:30:00.412365 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:00.412331    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fb5de8f7-ab6e-426e-9f58-e57de2f6acac-manifests\") pod \"opendatahub-operator-controller-manager-74fc54996-kqp5k\" (UID: \"fb5de8f7-ab6e-426e-9f58-e57de2f6acac\") " pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-kqp5k"
Sep 09 15:30:00.412365 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:00.412367    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb5de8f7-ab6e-426e-9f58-e57de2f6acac-tmp\") pod \"opendatahub-operator-controller-manager-74fc54996-kqp5k\" (UID: \"fb5de8f7-ab6e-426e-9f58-e57de2f6acac\") " pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-kqp5k"
Sep 09 15:30:00.412539 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:00.412410    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qfm5\" (UniqueName: \"kubernetes.io/projected/fb5de8f7-ab6e-426e-9f58-e57de2f6acac-kube-api-access-2qfm5\") pod \"opendatahub-operator-controller-manager-74fc54996-kqp5k\" (UID: \"fb5de8f7-ab6e-426e-9f58-e57de2f6acac\") " pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-kqp5k"
Sep 09 15:30:00.412539 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:00.412453    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fb5de8f7-ab6e-426e-9f58-e57de2f6acac-webhook-cert\") pod \"opendatahub-operator-controller-manager-74fc54996-kqp5k\" (UID: \"fb5de8f7-ab6e-426e-9f58-e57de2f6acac\") " pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-kqp5k"
Sep 09 15:30:00.412539 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:00.412499    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fb5de8f7-ab6e-426e-9f58-e57de2f6acac-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74fc54996-kqp5k\" (UID: \"fb5de8f7-ab6e-426e-9f58-e57de2f6acac\") " pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-kqp5k"
Sep 09 15:30:00.412865 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:00.412838    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fb5de8f7-ab6e-426e-9f58-e57de2f6acac-manifests\") pod \"opendatahub-operator-controller-manager-74fc54996-kqp5k\" (UID: \"fb5de8f7-ab6e-426e-9f58-e57de2f6acac\") " pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-kqp5k"
Sep 09 15:30:00.414723 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:00.414703    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb5de8f7-ab6e-426e-9f58-e57de2f6acac-tmp\") pod \"opendatahub-operator-controller-manager-74fc54996-kqp5k\" (UID: \"fb5de8f7-ab6e-426e-9f58-e57de2f6acac\") " pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-kqp5k"
Sep 09 15:30:00.415137 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:00.415116    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fb5de8f7-ab6e-426e-9f58-e57de2f6acac-webhook-cert\") pod \"opendatahub-operator-controller-manager-74fc54996-kqp5k\" (UID: \"fb5de8f7-ab6e-426e-9f58-e57de2f6acac\") " pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-kqp5k"
Sep 09 15:30:00.415194 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:00.415116    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fb5de8f7-ab6e-426e-9f58-e57de2f6acac-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74fc54996-kqp5k\" (UID: \"fb5de8f7-ab6e-426e-9f58-e57de2f6acac\") " pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-kqp5k"
Sep 09 15:30:00.421166 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:00.421135    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qfm5\" (UniqueName: \"kubernetes.io/projected/fb5de8f7-ab6e-426e-9f58-e57de2f6acac-kube-api-access-2qfm5\") pod \"opendatahub-operator-controller-manager-74fc54996-kqp5k\" (UID: \"fb5de8f7-ab6e-426e-9f58-e57de2f6acac\") " pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-kqp5k"
Sep 09 15:30:00.481125 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:00.481089    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-kqp5k"
Sep 09 15:30:00.758492 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:00.758418    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74fc54996-kqp5k"]
Sep 09 15:30:00.762178 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:30:00.762153    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb5de8f7_ab6e_426e_9f58_e57de2f6acac.slice/crio-e6a627ecfa2ba78da9341e3f85a5ce2c4a6a1750e17bb2db35607e9f34838aea WatchSource:0}: Error finding container e6a627ecfa2ba78da9341e3f85a5ce2c4a6a1750e17bb2db35607e9f34838aea: Status 404 returned error can't find the container with id e6a627ecfa2ba78da9341e3f85a5ce2c4a6a1750e17bb2db35607e9f34838aea
Sep 09 15:30:00.813220 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:00.813184    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-kqp5k" event={"ID":"fb5de8f7-ab6e-426e-9f58-e57de2f6acac","Type":"ContainerStarted","Data":"e6a627ecfa2ba78da9341e3f85a5ce2c4a6a1750e17bb2db35607e9f34838aea"}
Sep 09 15:30:06.839694 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:06.839651    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-kqp5k" event={"ID":"fb5de8f7-ab6e-426e-9f58-e57de2f6acac","Type":"ContainerStarted","Data":"0fc72752c82a51cc7aa40655b7a14cb112ed334e2f9ed3c67c877731d9d7456b"}
Sep 09 15:30:08.850124 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:08.850084    2553 generic.go:356] "Generic (PLEG): container finished" podID="fb5de8f7-ab6e-426e-9f58-e57de2f6acac" containerID="0fc72752c82a51cc7aa40655b7a14cb112ed334e2f9ed3c67c877731d9d7456b" exitCode=0
Sep 09 15:30:08.850612 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:08.850165    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-kqp5k" event={"ID":"fb5de8f7-ab6e-426e-9f58-e57de2f6acac","Type":"ContainerDied","Data":"0fc72752c82a51cc7aa40655b7a14cb112ed334e2f9ed3c67c877731d9d7456b"}
Sep 09 15:30:09.810756 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:09.810701    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-76f449f6db-2j5kb"
Sep 09 15:30:09.856351 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:09.856320    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-kqp5k" event={"ID":"fb5de8f7-ab6e-426e-9f58-e57de2f6acac","Type":"ContainerStarted","Data":"fc5fdb206f1da71ecaa3e8582ebb6c9b2d36381292efc6f80da461e46acf1864"}
Sep 09 15:30:09.856788 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:09.856605    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-kqp5k"
Sep 09 15:30:09.878167 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:09.878097    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-kqp5k" podStartSLOduration=2.560904999 podStartE2EDuration="9.878074077s" totalImagesPullingTime="6.227663681s" totalInitContainerRuntime="1.089505397s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:30:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:30:09.875987363 +0000 UTC m=+575.474882008" watchObservedRunningTime="2026-09-09 15:30:09.878074077 +0000 UTC m=+575.476968722"
Sep 09 15:30:20.863187 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:20.863143    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-kqp5k"
Sep 09 15:30:36.983895 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:36.983859    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9nb4x"]
Sep 09 15:30:36.986849 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:36.986828    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-9nb4x"
Sep 09 15:30:36.989447 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:36.989421    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 09 15:30:36.989607 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:36.989454    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-bm8ld\""
Sep 09 15:30:36.994226 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:36.994200    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9nb4x"]
Sep 09 15:30:37.083396 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:37.083349    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99e8b784-eaba-41c5-93b1-62683142a4c7-cert\") pod \"odh-model-controller-7998c67dc4-9nb4x\" (UID: \"99e8b784-eaba-41c5-93b1-62683142a4c7\") " pod="opendatahub/odh-model-controller-7998c67dc4-9nb4x"
Sep 09 15:30:37.083584 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:37.083467    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jls57\" (UniqueName: \"kubernetes.io/projected/99e8b784-eaba-41c5-93b1-62683142a4c7-kube-api-access-jls57\") pod \"odh-model-controller-7998c67dc4-9nb4x\" (UID: \"99e8b784-eaba-41c5-93b1-62683142a4c7\") " pod="opendatahub/odh-model-controller-7998c67dc4-9nb4x"
Sep 09 15:30:37.183925 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:37.183882    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99e8b784-eaba-41c5-93b1-62683142a4c7-cert\") pod \"odh-model-controller-7998c67dc4-9nb4x\" (UID: \"99e8b784-eaba-41c5-93b1-62683142a4c7\") " pod="opendatahub/odh-model-controller-7998c67dc4-9nb4x"
Sep 09 15:30:37.184132 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:37.184029    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jls57\" (UniqueName: \"kubernetes.io/projected/99e8b784-eaba-41c5-93b1-62683142a4c7-kube-api-access-jls57\") pod \"odh-model-controller-7998c67dc4-9nb4x\" (UID: \"99e8b784-eaba-41c5-93b1-62683142a4c7\") " pod="opendatahub/odh-model-controller-7998c67dc4-9nb4x"
Sep 09 15:30:37.184132 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:30:37.184047    2553 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 09 15:30:37.184132 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:30:37.184120    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e8b784-eaba-41c5-93b1-62683142a4c7-cert podName:99e8b784-eaba-41c5-93b1-62683142a4c7 nodeName:}" failed. No retries permitted until 2026-09-09 15:30:37.684097024 +0000 UTC m=+603.282991650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/99e8b784-eaba-41c5-93b1-62683142a4c7-cert") pod "odh-model-controller-7998c67dc4-9nb4x" (UID: "99e8b784-eaba-41c5-93b1-62683142a4c7") : secret "odh-model-controller-webhook-cert" not found
Sep 09 15:30:37.192812 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:37.192787    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jls57\" (UniqueName: \"kubernetes.io/projected/99e8b784-eaba-41c5-93b1-62683142a4c7-kube-api-access-jls57\") pod \"odh-model-controller-7998c67dc4-9nb4x\" (UID: \"99e8b784-eaba-41c5-93b1-62683142a4c7\") " pod="opendatahub/odh-model-controller-7998c67dc4-9nb4x"
Sep 09 15:30:37.580189 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:37.580154    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-p5sc6"]
Sep 09 15:30:37.582629 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:37.582612    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-p5sc6"
Sep 09 15:30:37.585283 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:37.585245    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 09 15:30:37.586973 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:37.586946    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnkwr\" (UniqueName: \"kubernetes.io/projected/1a6f2c0c-b12e-4e1a-9192-95987ff1047b-kube-api-access-wnkwr\") pod \"model-serving-api-54d89bd5fc-p5sc6\" (UID: \"1a6f2c0c-b12e-4e1a-9192-95987ff1047b\") " pod="opendatahub/model-serving-api-54d89bd5fc-p5sc6"
Sep 09 15:30:37.587083 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:37.586993    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a6f2c0c-b12e-4e1a-9192-95987ff1047b-tls-certs\") pod \"model-serving-api-54d89bd5fc-p5sc6\" (UID: \"1a6f2c0c-b12e-4e1a-9192-95987ff1047b\") " pod="opendatahub/model-serving-api-54d89bd5fc-p5sc6"
Sep 09 15:30:37.594810 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:37.594787    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-p5sc6"]
Sep 09 15:30:37.688338 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:37.688300    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnkwr\" (UniqueName: \"kubernetes.io/projected/1a6f2c0c-b12e-4e1a-9192-95987ff1047b-kube-api-access-wnkwr\") pod \"model-serving-api-54d89bd5fc-p5sc6\" (UID: \"1a6f2c0c-b12e-4e1a-9192-95987ff1047b\") " pod="opendatahub/model-serving-api-54d89bd5fc-p5sc6"
Sep 09 15:30:37.688492 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:37.688346    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a6f2c0c-b12e-4e1a-9192-95987ff1047b-tls-certs\") pod \"model-serving-api-54d89bd5fc-p5sc6\" (UID: \"1a6f2c0c-b12e-4e1a-9192-95987ff1047b\") " pod="opendatahub/model-serving-api-54d89bd5fc-p5sc6"
Sep 09 15:30:37.688492 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:37.688387    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99e8b784-eaba-41c5-93b1-62683142a4c7-cert\") pod \"odh-model-controller-7998c67dc4-9nb4x\" (UID: \"99e8b784-eaba-41c5-93b1-62683142a4c7\") " pod="opendatahub/odh-model-controller-7998c67dc4-9nb4x"
Sep 09 15:30:37.690904 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:37.690874    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a6f2c0c-b12e-4e1a-9192-95987ff1047b-tls-certs\") pod \"model-serving-api-54d89bd5fc-p5sc6\" (UID: \"1a6f2c0c-b12e-4e1a-9192-95987ff1047b\") " pod="opendatahub/model-serving-api-54d89bd5fc-p5sc6"
Sep 09 15:30:37.691025 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:37.690872    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99e8b784-eaba-41c5-93b1-62683142a4c7-cert\") pod \"odh-model-controller-7998c67dc4-9nb4x\" (UID: \"99e8b784-eaba-41c5-93b1-62683142a4c7\") " pod="opendatahub/odh-model-controller-7998c67dc4-9nb4x"
Sep 09 15:30:37.701348 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:37.701317    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnkwr\" (UniqueName: \"kubernetes.io/projected/1a6f2c0c-b12e-4e1a-9192-95987ff1047b-kube-api-access-wnkwr\") pod \"model-serving-api-54d89bd5fc-p5sc6\" (UID: \"1a6f2c0c-b12e-4e1a-9192-95987ff1047b\") " pod="opendatahub/model-serving-api-54d89bd5fc-p5sc6"
Sep 09 15:30:37.892425 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:37.892329    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-p5sc6"
Sep 09 15:30:37.897806 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:37.897778    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-9nb4x"
Sep 09 15:30:38.169075 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.169047    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-p5sc6"]
Sep 09 15:30:38.171704 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:30:38.171672    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a6f2c0c_b12e_4e1a_9192_95987ff1047b.slice/crio-7b41eda5cb01cfcc68649c5ca33ff562a30418cafd6570cd5033c71c30962999 WatchSource:0}: Error finding container 7b41eda5cb01cfcc68649c5ca33ff562a30418cafd6570cd5033c71c30962999: Status 404 returned error can't find the container with id 7b41eda5cb01cfcc68649c5ca33ff562a30418cafd6570cd5033c71c30962999
Sep 09 15:30:38.175602 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.175579    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9nb4x"]
Sep 09 15:30:38.178275 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:30:38.178244    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e8b784_eaba_41c5_93b1_62683142a4c7.slice/crio-29103395d80da0c908ad2c21ea0182073f9300260b42462f28a0a99612cfee58 WatchSource:0}: Error finding container 29103395d80da0c908ad2c21ea0182073f9300260b42462f28a0a99612cfee58: Status 404 returned error can't find the container with id 29103395d80da0c908ad2c21ea0182073f9300260b42462f28a0a99612cfee58
Sep 09 15:30:38.348921 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.348887    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9"]
Sep 09 15:30:38.352125 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.352104    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9"
Sep 09 15:30:38.357266 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.357242    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 15:30:38.357614 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.357593    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 15:30:38.357798 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.357776    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-ld27v\""
Sep 09 15:30:38.357877 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.357783    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 15:30:38.365203 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.365177    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9"]
Sep 09 15:30:38.394999 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.394961    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a600f2a-266f-4789-8462-ff24a72f853a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9\" (UID: \"9a600f2a-266f-4789-8462-ff24a72f853a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9"
Sep 09 15:30:38.394999 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.394995    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a600f2a-266f-4789-8462-ff24a72f853a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9\" (UID: \"9a600f2a-266f-4789-8462-ff24a72f853a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9"
Sep 09 15:30:38.395274 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.395015    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a600f2a-266f-4789-8462-ff24a72f853a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9\" (UID: \"9a600f2a-266f-4789-8462-ff24a72f853a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9"
Sep 09 15:30:38.395274 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.395121    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a600f2a-266f-4789-8462-ff24a72f853a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9\" (UID: \"9a600f2a-266f-4789-8462-ff24a72f853a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9"
Sep 09 15:30:38.395274 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.395162    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6jkt\" (UniqueName: \"kubernetes.io/projected/9a600f2a-266f-4789-8462-ff24a72f853a-kube-api-access-t6jkt\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9\" (UID: \"9a600f2a-266f-4789-8462-ff24a72f853a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9"
Sep 09 15:30:38.395274 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.395201    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a600f2a-266f-4789-8462-ff24a72f853a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9\" (UID: \"9a600f2a-266f-4789-8462-ff24a72f853a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9"
Sep 09 15:30:38.395274 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.395219    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a600f2a-266f-4789-8462-ff24a72f853a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9\" (UID: \"9a600f2a-266f-4789-8462-ff24a72f853a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9"
Sep 09 15:30:38.395274 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.395251    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a600f2a-266f-4789-8462-ff24a72f853a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9\" (UID: \"9a600f2a-266f-4789-8462-ff24a72f853a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9"
Sep 09 15:30:38.395274 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.395273    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a600f2a-266f-4789-8462-ff24a72f853a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9\" (UID: \"9a600f2a-266f-4789-8462-ff24a72f853a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9"
Sep 09 15:30:38.496521 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.496423    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a600f2a-266f-4789-8462-ff24a72f853a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9\" (UID: \"9a600f2a-266f-4789-8462-ff24a72f853a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9"
Sep 09 15:30:38.496521 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.496466    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6jkt\" (UniqueName: \"kubernetes.io/projected/9a600f2a-266f-4789-8462-ff24a72f853a-kube-api-access-t6jkt\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9\" (UID: \"9a600f2a-266f-4789-8462-ff24a72f853a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9"
Sep 09 15:30:38.496521 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.496504    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a600f2a-266f-4789-8462-ff24a72f853a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9\" (UID: \"9a600f2a-266f-4789-8462-ff24a72f853a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9"
Sep 09 15:30:38.496836 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.496532    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a600f2a-266f-4789-8462-ff24a72f853a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9\" (UID: \"9a600f2a-266f-4789-8462-ff24a72f853a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9"
Sep 09 15:30:38.496836 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.496571    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a600f2a-266f-4789-8462-ff24a72f853a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9\" (UID: \"9a600f2a-266f-4789-8462-ff24a72f853a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9"
Sep 09 15:30:38.496836 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.496601    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a600f2a-266f-4789-8462-ff24a72f853a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9\" (UID: \"9a600f2a-266f-4789-8462-ff24a72f853a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9"
Sep 09 15:30:38.496836 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.496634    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a600f2a-266f-4789-8462-ff24a72f853a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9\" (UID: \"9a600f2a-266f-4789-8462-ff24a72f853a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9"
Sep 09 15:30:38.496836 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.496656    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a600f2a-266f-4789-8462-ff24a72f853a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9\" (UID: \"9a600f2a-266f-4789-8462-ff24a72f853a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9"
Sep 09 15:30:38.496836 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.496682    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a600f2a-266f-4789-8462-ff24a72f853a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9\" (UID: \"9a600f2a-266f-4789-8462-ff24a72f853a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9"
Sep 09 15:30:38.497191 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.497136    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a600f2a-266f-4789-8462-ff24a72f853a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9\" (UID: \"9a600f2a-266f-4789-8462-ff24a72f853a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9"
Sep 09 15:30:38.497191 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.497166    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a600f2a-266f-4789-8462-ff24a72f853a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9\" (UID: \"9a600f2a-266f-4789-8462-ff24a72f853a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9"
Sep 09 15:30:38.497370 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.497282    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a600f2a-266f-4789-8462-ff24a72f853a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9\" (UID: \"9a600f2a-266f-4789-8462-ff24a72f853a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9"
Sep 09 15:30:38.497370 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.497318    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a600f2a-266f-4789-8462-ff24a72f853a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9\" (UID: \"9a600f2a-266f-4789-8462-ff24a72f853a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9"
Sep 09 15:30:38.497586 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.497565    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a600f2a-266f-4789-8462-ff24a72f853a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9\" (UID: \"9a600f2a-266f-4789-8462-ff24a72f853a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9"
Sep 09 15:30:38.498937 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.498912    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a600f2a-266f-4789-8462-ff24a72f853a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9\" (UID: \"9a600f2a-266f-4789-8462-ff24a72f853a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9"
Sep 09 15:30:38.499435 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.499417    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a600f2a-266f-4789-8462-ff24a72f853a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9\" (UID: \"9a600f2a-266f-4789-8462-ff24a72f853a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9"
Sep 09 15:30:38.505035 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.504979    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a600f2a-266f-4789-8462-ff24a72f853a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9\" (UID: \"9a600f2a-266f-4789-8462-ff24a72f853a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9"
Sep 09 15:30:38.505172 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.505149    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6jkt\" (UniqueName: \"kubernetes.io/projected/9a600f2a-266f-4789-8462-ff24a72f853a-kube-api-access-t6jkt\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9\" (UID: \"9a600f2a-266f-4789-8462-ff24a72f853a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9"
Sep 09 15:30:38.663660 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.663617    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9"
Sep 09 15:30:38.935576 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.935534    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9"]
Sep 09 15:30:38.938961 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:30:38.938939    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a600f2a_266f_4789_8462_ff24a72f853a.slice/crio-a2c6a1d24e93e7b6b0447b6de72f574177f96453ce29726cb46b07e54953b178 WatchSource:0}: Error finding container a2c6a1d24e93e7b6b0447b6de72f574177f96453ce29726cb46b07e54953b178: Status 404 returned error can't find the container with id a2c6a1d24e93e7b6b0447b6de72f574177f96453ce29726cb46b07e54953b178
Sep 09 15:30:38.962808 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.962766    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-9nb4x" event={"ID":"99e8b784-eaba-41c5-93b1-62683142a4c7","Type":"ContainerStarted","Data":"29103395d80da0c908ad2c21ea0182073f9300260b42462f28a0a99612cfee58"}
Sep 09 15:30:38.963627 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.963592    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9" event={"ID":"9a600f2a-266f-4789-8462-ff24a72f853a","Type":"ContainerStarted","Data":"a2c6a1d24e93e7b6b0447b6de72f574177f96453ce29726cb46b07e54953b178"}
Sep 09 15:30:38.964889 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:38.964868    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-p5sc6" event={"ID":"1a6f2c0c-b12e-4e1a-9192-95987ff1047b","Type":"ContainerStarted","Data":"7b41eda5cb01cfcc68649c5ca33ff562a30418cafd6570cd5033c71c30962999"}
Sep 09 15:30:45.153653 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:45.153609    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:30:45.153972 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:45.153704    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:30:45.153972 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:45.153756    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:30:46.003766 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:46.003717    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-9nb4x" event={"ID":"99e8b784-eaba-41c5-93b1-62683142a4c7","Type":"ContainerStarted","Data":"1c7c5f35202d1081b9bf549b90bbdce89bb0eb6e4f05310df950ebebc128a209"}
Sep 09 15:30:46.003977 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:46.003827    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-9nb4x"
Sep 09 15:30:46.005525 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:46.005497    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9" event={"ID":"9a600f2a-266f-4789-8462-ff24a72f853a","Type":"ContainerStarted","Data":"ae1194fb84b09395623a8c214c5ecb714752232dd63f09269c1dcc344e059bf5"}
Sep 09 15:30:46.039323 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:46.039254    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-9nb4x" podStartSLOduration=3.016598209 podStartE2EDuration="10.039229924s" totalImagesPullingTime="7.022631715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:30:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:30:46.035713346 +0000 UTC m=+611.634607992" watchObservedRunningTime="2026-09-09 15:30:46.039229924 +0000 UTC m=+611.638124572"
Sep 09 15:30:46.069950 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:46.069896    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9" podStartSLOduration=1.8575735199999999 podStartE2EDuration="8.069879383s" totalImagesPullingTime="6.212305863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:30:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:30:46.067672159 +0000 UTC m=+611.666566795" watchObservedRunningTime="2026-09-09 15:30:46.069879383 +0000 UTC m=+611.668774026"
Sep 09 15:30:46.664342 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:46.664301    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9"
Sep 09 15:30:46.669942 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:46.669913    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9"
Sep 09 15:30:47.012027 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:47.011629    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9"
Sep 09 15:30:47.013320 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:47.013293    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9"
Sep 09 15:30:48.016133 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:48.016097    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-p5sc6" event={"ID":"1a6f2c0c-b12e-4e1a-9192-95987ff1047b","Type":"ContainerStarted","Data":"954e170d615289824895607cacd24c7a2b4df23401a6ef3f885abc429970726a"}
Sep 09 15:30:48.039955 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:48.039899    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-p5sc6" podStartSLOduration=2.003454516 podStartE2EDuration="11.039882374s" totalImagesPullingTime="9.036427858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:30:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:30:48.03833696 +0000 UTC m=+613.637231606" watchObservedRunningTime="2026-09-09 15:30:48.039882374 +0000 UTC m=+613.638777018"
Sep 09 15:30:49.020451 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:49.020418    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-p5sc6"
Sep 09 15:30:57.014179 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:30:57.014099    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-9nb4x"
Sep 09 15:31:00.029266 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:31:00.029237    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-p5sc6"
Sep 09 15:31:58.478242 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:31:58.478203    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-75wp2"]
Sep 09 15:31:58.480444 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:31:58.480427    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-75wp2"
Sep 09 15:31:58.483772 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:31:58.483751    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-9pp5j\""
Sep 09 15:31:58.484496 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:31:58.484477    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 15:31:58.484588 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:31:58.484543    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 15:31:58.496162 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:31:58.496140    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-75wp2"]
Sep 09 15:31:58.537959 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:31:58.537921    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxb7j\" (UniqueName: \"kubernetes.io/projected/2b6a7f98-7526-4ec6-9d86-94413b11c76e-kube-api-access-cxb7j\") pod \"limitador-operator-controller-manager-74495c69f8-75wp2\" (UID: \"2b6a7f98-7526-4ec6-9d86-94413b11c76e\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-75wp2"
Sep 09 15:31:58.638981 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:31:58.638943    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxb7j\" (UniqueName: \"kubernetes.io/projected/2b6a7f98-7526-4ec6-9d86-94413b11c76e-kube-api-access-cxb7j\") pod \"limitador-operator-controller-manager-74495c69f8-75wp2\" (UID: \"2b6a7f98-7526-4ec6-9d86-94413b11c76e\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-75wp2"
Sep 09 15:31:58.651896 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:31:58.651864    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxb7j\" (UniqueName: \"kubernetes.io/projected/2b6a7f98-7526-4ec6-9d86-94413b11c76e-kube-api-access-cxb7j\") pod \"limitador-operator-controller-manager-74495c69f8-75wp2\" (UID: \"2b6a7f98-7526-4ec6-9d86-94413b11c76e\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-75wp2"
Sep 09 15:31:58.790608 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:31:58.790517    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-75wp2"
Sep 09 15:31:59.080537 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:31:59.080503    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-75wp2"]
Sep 09 15:31:59.081966 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:31:59.081930    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b6a7f98_7526_4ec6_9d86_94413b11c76e.slice/crio-c351d3cd98010a0b739b8d210e4a4bce531630ef6bcdb66b10bd80a5f699c11c WatchSource:0}: Error finding container c351d3cd98010a0b739b8d210e4a4bce531630ef6bcdb66b10bd80a5f699c11c: Status 404 returned error can't find the container with id c351d3cd98010a0b739b8d210e4a4bce531630ef6bcdb66b10bd80a5f699c11c
Sep 09 15:31:59.291663 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:31:59.291623    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-75wp2" event={"ID":"2b6a7f98-7526-4ec6-9d86-94413b11c76e","Type":"ContainerStarted","Data":"c351d3cd98010a0b739b8d210e4a4bce531630ef6bcdb66b10bd80a5f699c11c"}
Sep 09 15:32:06.322105 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:06.322062    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-75wp2" event={"ID":"2b6a7f98-7526-4ec6-9d86-94413b11c76e","Type":"ContainerStarted","Data":"7e6171fdb81dde81fca6794d697f16892bdcadf7775b693b8d58a72c2e35ddef"}
Sep 09 15:32:06.322488 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:06.322246    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-75wp2"
Sep 09 15:32:06.354807 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:06.354717    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-75wp2" podStartSLOduration=1.8511554110000001 podStartE2EDuration="8.354700084s" totalImagesPullingTime="6.503544673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:31:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:32:06.352650232 +0000 UTC m=+691.951544889" watchObservedRunningTime="2026-09-09 15:32:06.354700084 +0000 UTC m=+691.953594728"
Sep 09 15:32:17.328908 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:17.328870    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-75wp2"
Sep 09 15:32:19.034670 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:19.034628    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-wfjvz"]
Sep 09 15:32:19.036928 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:19.036906    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wfjvz"
Sep 09 15:32:19.039585 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:19.039562    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6768q\""
Sep 09 15:32:19.039722 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:19.039562    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 09 15:32:19.039722 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:19.039632    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 09 15:32:19.045644 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:19.045620    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-wfjvz"]
Sep 09 15:32:19.108615 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:19.108577    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5edfa589-8ed0-4951-a1d5-ea7ea08c126c-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-wfjvz\" (UID: \"5edfa589-8ed0-4951-a1d5-ea7ea08c126c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wfjvz"
Sep 09 15:32:19.108832 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:19.108779    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5edfa589-8ed0-4951-a1d5-ea7ea08c126c-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-wfjvz\" (UID: \"5edfa589-8ed0-4951-a1d5-ea7ea08c126c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wfjvz"
Sep 09 15:32:19.108832 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:19.108827    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zmhw\" (UniqueName: \"kubernetes.io/projected/5edfa589-8ed0-4951-a1d5-ea7ea08c126c-kube-api-access-2zmhw\") pod \"kuadrant-console-plugin-fbdd4f445-wfjvz\" (UID: \"5edfa589-8ed0-4951-a1d5-ea7ea08c126c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wfjvz"
Sep 09 15:32:19.209868 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:19.209822    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5edfa589-8ed0-4951-a1d5-ea7ea08c126c-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-wfjvz\" (UID: \"5edfa589-8ed0-4951-a1d5-ea7ea08c126c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wfjvz"
Sep 09 15:32:19.210033 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:19.209983    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5edfa589-8ed0-4951-a1d5-ea7ea08c126c-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-wfjvz\" (UID: \"5edfa589-8ed0-4951-a1d5-ea7ea08c126c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wfjvz"
Sep 09 15:32:19.210033 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:19.210022    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zmhw\" (UniqueName: \"kubernetes.io/projected/5edfa589-8ed0-4951-a1d5-ea7ea08c126c-kube-api-access-2zmhw\") pod \"kuadrant-console-plugin-fbdd4f445-wfjvz\" (UID: \"5edfa589-8ed0-4951-a1d5-ea7ea08c126c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wfjvz"
Sep 09 15:32:19.210142 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:32:19.210125    2553 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 09 15:32:19.210220 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:32:19.210209    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5edfa589-8ed0-4951-a1d5-ea7ea08c126c-plugin-serving-cert podName:5edfa589-8ed0-4951-a1d5-ea7ea08c126c nodeName:}" failed. No retries permitted until 2026-09-09 15:32:19.710184734 +0000 UTC m=+705.309079363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/5edfa589-8ed0-4951-a1d5-ea7ea08c126c-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-wfjvz" (UID: "5edfa589-8ed0-4951-a1d5-ea7ea08c126c") : secret "plugin-serving-cert" not found
Sep 09 15:32:19.210480 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:19.210459    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5edfa589-8ed0-4951-a1d5-ea7ea08c126c-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-wfjvz\" (UID: \"5edfa589-8ed0-4951-a1d5-ea7ea08c126c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wfjvz"
Sep 09 15:32:19.219264 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:19.219232    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zmhw\" (UniqueName: \"kubernetes.io/projected/5edfa589-8ed0-4951-a1d5-ea7ea08c126c-kube-api-access-2zmhw\") pod \"kuadrant-console-plugin-fbdd4f445-wfjvz\" (UID: \"5edfa589-8ed0-4951-a1d5-ea7ea08c126c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wfjvz"
Sep 09 15:32:19.715920 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:19.715871    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5edfa589-8ed0-4951-a1d5-ea7ea08c126c-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-wfjvz\" (UID: \"5edfa589-8ed0-4951-a1d5-ea7ea08c126c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wfjvz"
Sep 09 15:32:19.718341 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:19.718316    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5edfa589-8ed0-4951-a1d5-ea7ea08c126c-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-wfjvz\" (UID: \"5edfa589-8ed0-4951-a1d5-ea7ea08c126c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wfjvz"
Sep 09 15:32:19.946553 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:19.946512    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wfjvz"
Sep 09 15:32:20.217461 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:20.217430    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-wfjvz"]
Sep 09 15:32:20.220462 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:32:20.220430    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5edfa589_8ed0_4951_a1d5_ea7ea08c126c.slice/crio-a9ec69c3981075967657b91b7e11fc59e276256fccfa19903abf15a949baf92a WatchSource:0}: Error finding container a9ec69c3981075967657b91b7e11fc59e276256fccfa19903abf15a949baf92a: Status 404 returned error can't find the container with id a9ec69c3981075967657b91b7e11fc59e276256fccfa19903abf15a949baf92a
Sep 09 15:32:20.378305 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:20.378222    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wfjvz" event={"ID":"5edfa589-8ed0-4951-a1d5-ea7ea08c126c","Type":"ContainerStarted","Data":"a9ec69c3981075967657b91b7e11fc59e276256fccfa19903abf15a949baf92a"}
Sep 09 15:32:25.402977 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:25.402938    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wfjvz" event={"ID":"5edfa589-8ed0-4951-a1d5-ea7ea08c126c","Type":"ContainerStarted","Data":"eccda87fea843d8a568b0f40a6fae2630509155f94dc4827a2033c9803e2a92d"}
Sep 09 15:32:25.420327 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:25.420271    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wfjvz" podStartSLOduration=1.4391766449999999 podStartE2EDuration="6.420252775s" totalImagesPullingTime="4.98107613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:32:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:32:25.419452464 +0000 UTC m=+711.018347109" watchObservedRunningTime="2026-09-09 15:32:25.420252775 +0000 UTC m=+711.019147418"
Sep 09 15:32:29.824241 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:29.824203    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-75wp2"]
Sep 09 15:32:29.824797 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:29.824467    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-75wp2" podUID="2b6a7f98-7526-4ec6-9d86-94413b11c76e" containerName="manager" containerID="cri-o://7e6171fdb81dde81fca6794d697f16892bdcadf7775b693b8d58a72c2e35ddef" gracePeriod=2
Sep 09 15:32:29.832209 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:29.832178    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-75wp2"]
Sep 09 15:32:29.856089 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:29.854442    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-7zrwq"]
Sep 09 15:32:29.856261 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:29.856192    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b6a7f98-7526-4ec6-9d86-94413b11c76e" containerName="manager"
Sep 09 15:32:29.856261 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:29.856247    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b6a7f98-7526-4ec6-9d86-94413b11c76e" containerName="manager"
Sep 09 15:32:29.860409 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:29.856402    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b6a7f98-7526-4ec6-9d86-94413b11c76e" containerName="manager"
Sep 09 15:32:29.860564 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:29.860455    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7zrwq"
Sep 09 15:32:29.870343 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:29.870312    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-7zrwq"]
Sep 09 15:32:29.887936 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:32:29.887887    2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-75wp2\" is forbidden: User \"system:node:ip-10-0-129-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-170.ec2.internal' and this object" podUID="2b6a7f98-7526-4ec6-9d86-94413b11c76e" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-75wp2"
Sep 09 15:32:29.908910 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:32:29.908883    2553 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode44aa66f_7b4b_48f4_b384_865e874ff99e.slice\": RecentStats: unable to find data in memory cache]"
Sep 09 15:32:30.008240 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:30.008206    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgfbg\" (UniqueName: \"kubernetes.io/projected/e44aa66f-7b4b-48f4-b384-865e874ff99e-kube-api-access-pgfbg\") pod \"limitador-operator-controller-manager-74495c69f8-7zrwq\" (UID: \"e44aa66f-7b4b-48f4-b384-865e874ff99e\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7zrwq"
Sep 09 15:32:30.063207 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:30.063183    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-75wp2"
Sep 09 15:32:30.065519 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:32:30.065488    2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-75wp2\" is forbidden: User \"system:node:ip-10-0-129-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-170.ec2.internal' and this object" podUID="2b6a7f98-7526-4ec6-9d86-94413b11c76e" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-75wp2"
Sep 09 15:32:30.109550 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:30.109464    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgfbg\" (UniqueName: \"kubernetes.io/projected/e44aa66f-7b4b-48f4-b384-865e874ff99e-kube-api-access-pgfbg\") pod \"limitador-operator-controller-manager-74495c69f8-7zrwq\" (UID: \"e44aa66f-7b4b-48f4-b384-865e874ff99e\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7zrwq"
Sep 09 15:32:30.120093 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:30.120059    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgfbg\" (UniqueName: \"kubernetes.io/projected/e44aa66f-7b4b-48f4-b384-865e874ff99e-kube-api-access-pgfbg\") pod \"limitador-operator-controller-manager-74495c69f8-7zrwq\" (UID: \"e44aa66f-7b4b-48f4-b384-865e874ff99e\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7zrwq"
Sep 09 15:32:30.210299 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:30.210262    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxb7j\" (UniqueName: \"kubernetes.io/projected/2b6a7f98-7526-4ec6-9d86-94413b11c76e-kube-api-access-cxb7j\") pod \"2b6a7f98-7526-4ec6-9d86-94413b11c76e\" (UID: \"2b6a7f98-7526-4ec6-9d86-94413b11c76e\") "
Sep 09 15:32:30.212497 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:30.212475    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7zrwq"
Sep 09 15:32:30.212613 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:30.212476    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6a7f98-7526-4ec6-9d86-94413b11c76e-kube-api-access-cxb7j" (OuterVolumeSpecName: "kube-api-access-cxb7j") pod "2b6a7f98-7526-4ec6-9d86-94413b11c76e" (UID: "2b6a7f98-7526-4ec6-9d86-94413b11c76e"). InnerVolumeSpecName "kube-api-access-cxb7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:32:30.311321 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:30.311275    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxb7j\" (UniqueName: \"kubernetes.io/projected/2b6a7f98-7526-4ec6-9d86-94413b11c76e-kube-api-access-cxb7j\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:32:30.425143 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:30.425108    2553 generic.go:356] "Generic (PLEG): container finished" podID="2b6a7f98-7526-4ec6-9d86-94413b11c76e" containerID="7e6171fdb81dde81fca6794d697f16892bdcadf7775b693b8d58a72c2e35ddef" exitCode=0
Sep 09 15:32:30.425294 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:30.425159    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-75wp2"
Sep 09 15:32:30.425294 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:30.425212    2553 scope.go:117] "RemoveContainer" containerID="7e6171fdb81dde81fca6794d697f16892bdcadf7775b693b8d58a72c2e35ddef"
Sep 09 15:32:30.427946 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:32:30.427918    2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-75wp2\" is forbidden: User \"system:node:ip-10-0-129-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-170.ec2.internal' and this object" podUID="2b6a7f98-7526-4ec6-9d86-94413b11c76e" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-75wp2"
Sep 09 15:32:30.433925 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:30.433904    2553 scope.go:117] "RemoveContainer" containerID="7e6171fdb81dde81fca6794d697f16892bdcadf7775b693b8d58a72c2e35ddef"
Sep 09 15:32:30.434221 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:32:30.434202    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e6171fdb81dde81fca6794d697f16892bdcadf7775b693b8d58a72c2e35ddef\": container with ID starting with 7e6171fdb81dde81fca6794d697f16892bdcadf7775b693b8d58a72c2e35ddef not found: ID does not exist" containerID="7e6171fdb81dde81fca6794d697f16892bdcadf7775b693b8d58a72c2e35ddef"
Sep 09 15:32:30.434287 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:30.434233    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e6171fdb81dde81fca6794d697f16892bdcadf7775b693b8d58a72c2e35ddef"} err="failed to get container status \"7e6171fdb81dde81fca6794d697f16892bdcadf7775b693b8d58a72c2e35ddef\": rpc error: code = NotFound desc = could not find container \"7e6171fdb81dde81fca6794d697f16892bdcadf7775b693b8d58a72c2e35ddef\": container with ID starting with 7e6171fdb81dde81fca6794d697f16892bdcadf7775b693b8d58a72c2e35ddef not found: ID does not exist"
Sep 09 15:32:30.435686 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:32:30.435663    2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-75wp2\" is forbidden: User \"system:node:ip-10-0-129-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-170.ec2.internal' and this object" podUID="2b6a7f98-7526-4ec6-9d86-94413b11c76e" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-75wp2"
Sep 09 15:32:30.494938 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:30.494914    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-7zrwq"]
Sep 09 15:32:30.497370 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:32:30.497343    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode44aa66f_7b4b_48f4_b384_865e874ff99e.slice/crio-bbc1c36fd30988a723cebca6259e6a24a7021c6c7b7d486d3f72e6aaac93e7e9 WatchSource:0}: Error finding container bbc1c36fd30988a723cebca6259e6a24a7021c6c7b7d486d3f72e6aaac93e7e9: Status 404 returned error can't find the container with id bbc1c36fd30988a723cebca6259e6a24a7021c6c7b7d486d3f72e6aaac93e7e9
Sep 09 15:32:30.962163 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:30.962128    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b6a7f98-7526-4ec6-9d86-94413b11c76e" path="/var/lib/kubelet/pods/2b6a7f98-7526-4ec6-9d86-94413b11c76e/volumes"
Sep 09 15:32:31.432006 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:31.431973    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7zrwq" event={"ID":"e44aa66f-7b4b-48f4-b384-865e874ff99e","Type":"ContainerStarted","Data":"4caea906b1948b6f6f3633a142f0d25262be63657e08c043948c94ac07346b22"}
Sep 09 15:32:31.432208 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:31.432023    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7zrwq" event={"ID":"e44aa66f-7b4b-48f4-b384-865e874ff99e","Type":"ContainerStarted","Data":"bbc1c36fd30988a723cebca6259e6a24a7021c6c7b7d486d3f72e6aaac93e7e9"}
Sep 09 15:32:31.432208 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:31.432084    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7zrwq"
Sep 09 15:32:31.449669 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:31.449616    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7zrwq" podStartSLOduration=2.449602323 podStartE2EDuration="2.449602323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:32:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:32:31.448076865 +0000 UTC m=+717.046971508" watchObservedRunningTime="2026-09-09 15:32:31.449602323 +0000 UTC m=+717.048496966"
Sep 09 15:32:42.440034 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:42.440003    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7zrwq"
Sep 09 15:32:45.696913 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:45.696875    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-spk8r"]
Sep 09 15:32:45.701668 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:45.701641    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-spk8r"
Sep 09 15:32:45.704416 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:45.704392    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-4zwbk\""
Sep 09 15:32:45.714272 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:45.714239    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-spk8r"]
Sep 09 15:32:45.739535 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:45.739493    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxqkf\" (UniqueName: \"kubernetes.io/projected/ea8cd68f-0037-4e71-b169-1dd4bc0ba283-kube-api-access-nxqkf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-spk8r\" (UID: \"ea8cd68f-0037-4e71-b169-1dd4bc0ba283\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-spk8r"
Sep 09 15:32:45.739760 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:45.739553    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea8cd68f-0037-4e71-b169-1dd4bc0ba283-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-spk8r\" (UID: \"ea8cd68f-0037-4e71-b169-1dd4bc0ba283\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-spk8r"
Sep 09 15:32:45.840189 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:45.840127    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxqkf\" (UniqueName: \"kubernetes.io/projected/ea8cd68f-0037-4e71-b169-1dd4bc0ba283-kube-api-access-nxqkf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-spk8r\" (UID: \"ea8cd68f-0037-4e71-b169-1dd4bc0ba283\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-spk8r"
Sep 09 15:32:45.840189 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:45.840200    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea8cd68f-0037-4e71-b169-1dd4bc0ba283-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-spk8r\" (UID: \"ea8cd68f-0037-4e71-b169-1dd4bc0ba283\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-spk8r"
Sep 09 15:32:45.840550 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:45.840534    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea8cd68f-0037-4e71-b169-1dd4bc0ba283-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-spk8r\" (UID: \"ea8cd68f-0037-4e71-b169-1dd4bc0ba283\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-spk8r"
Sep 09 15:32:45.851136 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:45.851108    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxqkf\" (UniqueName: \"kubernetes.io/projected/ea8cd68f-0037-4e71-b169-1dd4bc0ba283-kube-api-access-nxqkf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-spk8r\" (UID: \"ea8cd68f-0037-4e71-b169-1dd4bc0ba283\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-spk8r"
Sep 09 15:32:46.011352 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:46.011264    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-spk8r"
Sep 09 15:32:46.295411 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:46.295387    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-spk8r"]
Sep 09 15:32:46.297789 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:32:46.297759    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea8cd68f_0037_4e71_b169_1dd4bc0ba283.slice/crio-751dfdb61fa9b02a3d7ff867c507217f38b3d9d5b2becbceb203dcd7af8e9211 WatchSource:0}: Error finding container 751dfdb61fa9b02a3d7ff867c507217f38b3d9d5b2becbceb203dcd7af8e9211: Status 404 returned error can't find the container with id 751dfdb61fa9b02a3d7ff867c507217f38b3d9d5b2becbceb203dcd7af8e9211
Sep 09 15:32:46.500366 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:46.500315    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-spk8r" event={"ID":"ea8cd68f-0037-4e71-b169-1dd4bc0ba283","Type":"ContainerStarted","Data":"751dfdb61fa9b02a3d7ff867c507217f38b3d9d5b2becbceb203dcd7af8e9211"}
Sep 09 15:32:50.521503 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:50.521460    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-spk8r" event={"ID":"ea8cd68f-0037-4e71-b169-1dd4bc0ba283","Type":"ContainerStarted","Data":"69ef1d919a2460b60fd9d116bdd2b00b0113cda6daf3ac8168c6ed64b01c7696"}
Sep 09 15:32:50.521956 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:50.521550    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-spk8r"
Sep 09 15:32:50.542722 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:32:50.542673    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-spk8r" podStartSLOduration=2.139559813 podStartE2EDuration="5.542654882s" totalImagesPullingTime="3.403095069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:32:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:32:50.54028448 +0000 UTC m=+736.139179125" watchObservedRunningTime="2026-09-09 15:32:50.542654882 +0000 UTC m=+736.141549587"
Sep 09 15:33:01.528138 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:33:01.528101    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-spk8r"
Sep 09 15:33:22.070963 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:33:22.070928    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-gtzw8"]
Sep 09 15:33:22.074151 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:33:22.074133    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-gtzw8"
Sep 09 15:33:22.076665 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:33:22.076645    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9frgw\""
Sep 09 15:33:22.081431 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:33:22.081056    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-gtzw8"]
Sep 09 15:33:22.131851 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:33:22.131818    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxx9t\" (UniqueName: \"kubernetes.io/projected/661f92a9-40dd-4a5e-8137-f5fb58aff303-kube-api-access-gxx9t\") pod \"authorino-59c974bd48-gtzw8\" (UID: \"661f92a9-40dd-4a5e-8137-f5fb58aff303\") " pod="kuadrant-system/authorino-59c974bd48-gtzw8"
Sep 09 15:33:22.232602 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:33:22.232561    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxx9t\" (UniqueName: \"kubernetes.io/projected/661f92a9-40dd-4a5e-8137-f5fb58aff303-kube-api-access-gxx9t\") pod \"authorino-59c974bd48-gtzw8\" (UID: \"661f92a9-40dd-4a5e-8137-f5fb58aff303\") " pod="kuadrant-system/authorino-59c974bd48-gtzw8"
Sep 09 15:33:22.241352 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:33:22.241318    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxx9t\" (UniqueName: \"kubernetes.io/projected/661f92a9-40dd-4a5e-8137-f5fb58aff303-kube-api-access-gxx9t\") pod \"authorino-59c974bd48-gtzw8\" (UID: \"661f92a9-40dd-4a5e-8137-f5fb58aff303\") " pod="kuadrant-system/authorino-59c974bd48-gtzw8"
Sep 09 15:33:22.384411 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:33:22.384320    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-gtzw8"
Sep 09 15:33:22.659609 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:33:22.659540    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-gtzw8"]
Sep 09 15:33:22.661845 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:33:22.661821    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod661f92a9_40dd_4a5e_8137_f5fb58aff303.slice/crio-65c04c20d58c70d74d7796f20db47b5b0726dfabac1051a9d0c0a0dd14280d1b WatchSource:0}: Error finding container 65c04c20d58c70d74d7796f20db47b5b0726dfabac1051a9d0c0a0dd14280d1b: Status 404 returned error can't find the container with id 65c04c20d58c70d74d7796f20db47b5b0726dfabac1051a9d0c0a0dd14280d1b
Sep 09 15:33:23.659465 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:33:23.659428    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-gtzw8" event={"ID":"661f92a9-40dd-4a5e-8137-f5fb58aff303","Type":"ContainerStarted","Data":"65c04c20d58c70d74d7796f20db47b5b0726dfabac1051a9d0c0a0dd14280d1b"}
Sep 09 15:33:25.668835 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:33:25.668798    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-gtzw8" event={"ID":"661f92a9-40dd-4a5e-8137-f5fb58aff303","Type":"ContainerStarted","Data":"25accc87029e3121c61724747bd70ef15e9b681a4142513cdad4d03733119478"}
Sep 09 15:33:25.685230 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:33:25.685174    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-gtzw8" podStartSLOduration=1.465017396 podStartE2EDuration="3.68515956s" totalImagesPullingTime="2.220142164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:33:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:33:25.683655053 +0000 UTC m=+771.282549698" watchObservedRunningTime="2026-09-09 15:33:25.68515956 +0000 UTC m=+771.284054203"
Sep 09 15:33:57.244198 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:33:57.244120    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-gtzw8"]
Sep 09 15:33:57.244690 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:33:57.244292    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-gtzw8" podUID="661f92a9-40dd-4a5e-8137-f5fb58aff303" containerName="authorino" containerID="cri-o://25accc87029e3121c61724747bd70ef15e9b681a4142513cdad4d03733119478" gracePeriod=30
Sep 09 15:33:57.498393 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:33:57.498324    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-gtzw8"
Sep 09 15:33:57.523951 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:33:57.523917    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxx9t\" (UniqueName: \"kubernetes.io/projected/661f92a9-40dd-4a5e-8137-f5fb58aff303-kube-api-access-gxx9t\") pod \"661f92a9-40dd-4a5e-8137-f5fb58aff303\" (UID: \"661f92a9-40dd-4a5e-8137-f5fb58aff303\") "
Sep 09 15:33:57.526053 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:33:57.526013    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/661f92a9-40dd-4a5e-8137-f5fb58aff303-kube-api-access-gxx9t" (OuterVolumeSpecName: "kube-api-access-gxx9t") pod "661f92a9-40dd-4a5e-8137-f5fb58aff303" (UID: "661f92a9-40dd-4a5e-8137-f5fb58aff303"). InnerVolumeSpecName "kube-api-access-gxx9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:33:57.624606 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:33:57.624561    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxx9t\" (UniqueName: \"kubernetes.io/projected/661f92a9-40dd-4a5e-8137-f5fb58aff303-kube-api-access-gxx9t\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:33:57.810787 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:33:57.810677    2553 generic.go:356] "Generic (PLEG): container finished" podID="661f92a9-40dd-4a5e-8137-f5fb58aff303" containerID="25accc87029e3121c61724747bd70ef15e9b681a4142513cdad4d03733119478" exitCode=0
Sep 09 15:33:57.810787 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:33:57.810723    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-gtzw8"
Sep 09 15:33:57.810787 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:33:57.810768    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-gtzw8" event={"ID":"661f92a9-40dd-4a5e-8137-f5fb58aff303","Type":"ContainerDied","Data":"25accc87029e3121c61724747bd70ef15e9b681a4142513cdad4d03733119478"}
Sep 09 15:33:57.811011 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:33:57.810811    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-gtzw8" event={"ID":"661f92a9-40dd-4a5e-8137-f5fb58aff303","Type":"ContainerDied","Data":"65c04c20d58c70d74d7796f20db47b5b0726dfabac1051a9d0c0a0dd14280d1b"}
Sep 09 15:33:57.811011 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:33:57.810829    2553 scope.go:117] "RemoveContainer" containerID="25accc87029e3121c61724747bd70ef15e9b681a4142513cdad4d03733119478"
Sep 09 15:33:57.819195 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:33:57.819177    2553 scope.go:117] "RemoveContainer" containerID="25accc87029e3121c61724747bd70ef15e9b681a4142513cdad4d03733119478"
Sep 09 15:33:57.819476 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:33:57.819457    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25accc87029e3121c61724747bd70ef15e9b681a4142513cdad4d03733119478\": container with ID starting with 25accc87029e3121c61724747bd70ef15e9b681a4142513cdad4d03733119478 not found: ID does not exist" containerID="25accc87029e3121c61724747bd70ef15e9b681a4142513cdad4d03733119478"
Sep 09 15:33:57.819536 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:33:57.819491    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25accc87029e3121c61724747bd70ef15e9b681a4142513cdad4d03733119478"} err="failed to get container status \"25accc87029e3121c61724747bd70ef15e9b681a4142513cdad4d03733119478\": rpc error: code = NotFound desc = could not find container \"25accc87029e3121c61724747bd70ef15e9b681a4142513cdad4d03733119478\": container with ID starting with 25accc87029e3121c61724747bd70ef15e9b681a4142513cdad4d03733119478 not found: ID does not exist"
Sep 09 15:33:57.837877 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:33:57.837848    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-gtzw8"]
Sep 09 15:33:57.843806 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:33:57.843779    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-gtzw8"]
Sep 09 15:33:58.961895 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:33:58.961858    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="661f92a9-40dd-4a5e-8137-f5fb58aff303" path="/var/lib/kubelet/pods/661f92a9-40dd-4a5e-8137-f5fb58aff303/volumes"
Sep 09 15:34:41.199000 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:34:41.198961    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-xjtqr"]
Sep 09 15:34:41.199409 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:34:41.199254    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="661f92a9-40dd-4a5e-8137-f5fb58aff303" containerName="authorino"
Sep 09 15:34:41.199409 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:34:41.199266    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="661f92a9-40dd-4a5e-8137-f5fb58aff303" containerName="authorino"
Sep 09 15:34:41.199409 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:34:41.199324    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="661f92a9-40dd-4a5e-8137-f5fb58aff303" containerName="authorino"
Sep 09 15:34:41.202550 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:34:41.202532    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xjtqr"
Sep 09 15:34:41.205473 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:34:41.205449    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 09 15:34:41.205613 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:34:41.205449    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-c5xnn\""
Sep 09 15:34:41.213914 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:34:41.213889    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-xjtqr"]
Sep 09 15:34:41.300474 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:34:41.300437    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/785531c7-305c-49be-b54f-4c0a6d598df3-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-xjtqr\" (UID: \"785531c7-305c-49be-b54f-4c0a6d598df3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xjtqr"
Sep 09 15:34:41.300649 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:34:41.300489    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qsz2\" (UniqueName: \"kubernetes.io/projected/785531c7-305c-49be-b54f-4c0a6d598df3-kube-api-access-4qsz2\") pod \"payload-pre-processing-59fbc47bf8-xjtqr\" (UID: \"785531c7-305c-49be-b54f-4c0a6d598df3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xjtqr"
Sep 09 15:34:41.401309 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:34:41.401269    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/785531c7-305c-49be-b54f-4c0a6d598df3-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-xjtqr\" (UID: \"785531c7-305c-49be-b54f-4c0a6d598df3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xjtqr"
Sep 09 15:34:41.401498 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:34:41.401324    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qsz2\" (UniqueName: \"kubernetes.io/projected/785531c7-305c-49be-b54f-4c0a6d598df3-kube-api-access-4qsz2\") pod \"payload-pre-processing-59fbc47bf8-xjtqr\" (UID: \"785531c7-305c-49be-b54f-4c0a6d598df3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xjtqr"
Sep 09 15:34:41.401990 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:34:41.401970    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/785531c7-305c-49be-b54f-4c0a6d598df3-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-xjtqr\" (UID: \"785531c7-305c-49be-b54f-4c0a6d598df3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xjtqr"
Sep 09 15:34:41.409080 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:34:41.409057    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qsz2\" (UniqueName: \"kubernetes.io/projected/785531c7-305c-49be-b54f-4c0a6d598df3-kube-api-access-4qsz2\") pod \"payload-pre-processing-59fbc47bf8-xjtqr\" (UID: \"785531c7-305c-49be-b54f-4c0a6d598df3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xjtqr"
Sep 09 15:34:41.514171 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:34:41.514077    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xjtqr"
Sep 09 15:34:41.787143 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:34:41.787083    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-xjtqr"]
Sep 09 15:34:41.789889 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:34:41.789862    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod785531c7_305c_49be_b54f_4c0a6d598df3.slice/crio-02ff41920f34373dc4af688487d6f436d3c970fc52fd8649e123ca01be6caea3 WatchSource:0}: Error finding container 02ff41920f34373dc4af688487d6f436d3c970fc52fd8649e123ca01be6caea3: Status 404 returned error can't find the container with id 02ff41920f34373dc4af688487d6f436d3c970fc52fd8649e123ca01be6caea3
Sep 09 15:34:41.792152 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:34:41.792132    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 15:34:41.998413 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:34:41.998377    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xjtqr" event={"ID":"785531c7-305c-49be-b54f-4c0a6d598df3","Type":"ContainerStarted","Data":"02ff41920f34373dc4af688487d6f436d3c970fc52fd8649e123ca01be6caea3"}
Sep 09 15:34:45.012247 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:34:45.012207    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xjtqr" event={"ID":"785531c7-305c-49be-b54f-4c0a6d598df3","Type":"ContainerStarted","Data":"278c9c8080546e0c9113764b81940ba1779841a575b14fc7c6c2006ccb324791"}
Sep 09 15:34:45.012705 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:34:45.012391    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xjtqr"
Sep 09 15:34:45.029056 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:34:45.029002    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xjtqr" podStartSLOduration=1.6008978790000001 podStartE2EDuration="4.028986402s" totalImagesPullingTime="2.428088523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:34:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:34:45.02765687 +0000 UTC m=+850.626551514" watchObservedRunningTime="2026-09-09 15:34:45.028986402 +0000 UTC m=+850.627881046"
Sep 09 15:34:56.018633 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:34:56.018599    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xjtqr"
Sep 09 15:35:14.850297 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:35:14.850259    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-6g8lw"]
Sep 09 15:35:14.860326 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:35:14.860288    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6g8lw"
Sep 09 15:35:14.861979 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:35:14.861949    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-6g8lw"]
Sep 09 15:35:14.864204 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:35:14.864184    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 09 15:35:14.864319 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:35:14.864221    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dz98x\""
Sep 09 15:35:14.864420 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:35:14.864398    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 09 15:35:14.864483 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:35:14.864434    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 09 15:35:14.968321 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:35:14.968278    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/163dac45-745e-4a6e-ac4b-8eb12a7925bb-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6g8lw\" (UID: \"163dac45-745e-4a6e-ac4b-8eb12a7925bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6g8lw"
Sep 09 15:35:14.968321 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:35:14.968322    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snqbt\" (UniqueName: \"kubernetes.io/projected/163dac45-745e-4a6e-ac4b-8eb12a7925bb-kube-api-access-snqbt\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6g8lw\" (UID: \"163dac45-745e-4a6e-ac4b-8eb12a7925bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6g8lw"
Sep 09 15:35:14.968551 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:35:14.968356    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/163dac45-745e-4a6e-ac4b-8eb12a7925bb-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6g8lw\" (UID: \"163dac45-745e-4a6e-ac4b-8eb12a7925bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6g8lw"
Sep 09 15:35:14.968551 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:35:14.968448    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/163dac45-745e-4a6e-ac4b-8eb12a7925bb-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6g8lw\" (UID: \"163dac45-745e-4a6e-ac4b-8eb12a7925bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6g8lw"
Sep 09 15:35:14.968551 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:35:14.968490    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/163dac45-745e-4a6e-ac4b-8eb12a7925bb-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6g8lw\" (UID: \"163dac45-745e-4a6e-ac4b-8eb12a7925bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6g8lw"
Sep 09 15:35:14.968551 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:35:14.968525    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/163dac45-745e-4a6e-ac4b-8eb12a7925bb-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6g8lw\" (UID: \"163dac45-745e-4a6e-ac4b-8eb12a7925bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6g8lw"
Sep 09 15:35:15.069815 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:35:15.069778    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/163dac45-745e-4a6e-ac4b-8eb12a7925bb-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6g8lw\" (UID: \"163dac45-745e-4a6e-ac4b-8eb12a7925bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6g8lw"
Sep 09 15:35:15.069996 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:35:15.069919    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/163dac45-745e-4a6e-ac4b-8eb12a7925bb-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6g8lw\" (UID: \"163dac45-745e-4a6e-ac4b-8eb12a7925bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6g8lw"
Sep 09 15:35:15.069996 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:35:15.069957    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snqbt\" (UniqueName: \"kubernetes.io/projected/163dac45-745e-4a6e-ac4b-8eb12a7925bb-kube-api-access-snqbt\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6g8lw\" (UID: \"163dac45-745e-4a6e-ac4b-8eb12a7925bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6g8lw"
Sep 09 15:35:15.069996 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:35:15.069981    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/163dac45-745e-4a6e-ac4b-8eb12a7925bb-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6g8lw\" (UID: \"163dac45-745e-4a6e-ac4b-8eb12a7925bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6g8lw"
Sep 09 15:35:15.070156 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:35:15.070138    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/163dac45-745e-4a6e-ac4b-8eb12a7925bb-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6g8lw\" (UID: \"163dac45-745e-4a6e-ac4b-8eb12a7925bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6g8lw"
Sep 09 15:35:15.070227 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:35:15.070205    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/163dac45-745e-4a6e-ac4b-8eb12a7925bb-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6g8lw\" (UID: \"163dac45-745e-4a6e-ac4b-8eb12a7925bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6g8lw"
Sep 09 15:35:15.070285 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:35:15.070234    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/163dac45-745e-4a6e-ac4b-8eb12a7925bb-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6g8lw\" (UID: \"163dac45-745e-4a6e-ac4b-8eb12a7925bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6g8lw"
Sep 09 15:35:15.070402 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:35:15.070362    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/163dac45-745e-4a6e-ac4b-8eb12a7925bb-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6g8lw\" (UID: \"163dac45-745e-4a6e-ac4b-8eb12a7925bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6g8lw"
Sep 09 15:35:15.070509 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:35:15.070471    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/163dac45-745e-4a6e-ac4b-8eb12a7925bb-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6g8lw\" (UID: \"163dac45-745e-4a6e-ac4b-8eb12a7925bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6g8lw"
Sep 09 15:35:15.072387 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:35:15.072360    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/163dac45-745e-4a6e-ac4b-8eb12a7925bb-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6g8lw\" (UID: \"163dac45-745e-4a6e-ac4b-8eb12a7925bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6g8lw"
Sep 09 15:35:15.072688 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:35:15.072668    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/163dac45-745e-4a6e-ac4b-8eb12a7925bb-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6g8lw\" (UID: \"163dac45-745e-4a6e-ac4b-8eb12a7925bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6g8lw"
Sep 09 15:35:15.079145 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:35:15.079110    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snqbt\" (UniqueName: \"kubernetes.io/projected/163dac45-745e-4a6e-ac4b-8eb12a7925bb-kube-api-access-snqbt\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6g8lw\" (UID: \"163dac45-745e-4a6e-ac4b-8eb12a7925bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6g8lw"
Sep 09 15:35:15.170618 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:35:15.170577    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6g8lw"
Sep 09 15:35:15.443675 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:35:15.443645    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-6g8lw"]
Sep 09 15:35:15.446123 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:35:15.446096    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod163dac45_745e_4a6e_ac4b_8eb12a7925bb.slice/crio-f9190328f9a16bca82a7d0635138060d6067b7ab4bb7091f749dbe41f534fcea WatchSource:0}: Error finding container f9190328f9a16bca82a7d0635138060d6067b7ab4bb7091f749dbe41f534fcea: Status 404 returned error can't find the container with id f9190328f9a16bca82a7d0635138060d6067b7ab4bb7091f749dbe41f534fcea
Sep 09 15:35:16.144692 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:35:16.144654    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6g8lw" event={"ID":"163dac45-745e-4a6e-ac4b-8eb12a7925bb","Type":"ContainerStarted","Data":"f9190328f9a16bca82a7d0635138060d6067b7ab4bb7091f749dbe41f534fcea"}
Sep 09 15:35:17.151092 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:35:17.151003    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6g8lw" event={"ID":"163dac45-745e-4a6e-ac4b-8eb12a7925bb","Type":"ContainerStarted","Data":"940cf8699bb22f9f6da93737624fabd3d2bd833a41ef80a080a654ff5d1c380b"}
Sep 09 15:35:17.173509 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:35:17.173457    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6g8lw" podStartSLOduration=2.187971426 podStartE2EDuration="3.173440441s" totalImagesPullingTime="985.469015ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:35:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:35:17.16925836 +0000 UTC m=+882.768153014" watchObservedRunningTime="2026-09-09 15:35:17.173440441 +0000 UTC m=+882.772335085"
Sep 09 15:35:25.171094 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:35:25.171049    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6g8lw"
Sep 09 15:35:25.171514 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:35:25.171124    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6g8lw"
Sep 09 15:35:25.183478 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:35:25.183441    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6g8lw"
Sep 09 15:35:25.195882 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:35:25.195851    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6g8lw"
Sep 09 15:36:15.561447 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:36:15.561414    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6"]
Sep 09 15:36:15.563379 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:36:15.563364    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6"
Sep 09 15:36:15.566160 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:36:15.566135    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 09 15:36:15.575231 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:36:15.575206    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6"]
Sep 09 15:36:15.691357 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:36:15.691300    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2ec6829-400f-42cc-bb98-f1f084ce16f9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6\" (UID: \"c2ec6829-400f-42cc-bb98-f1f084ce16f9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6"
Sep 09 15:36:15.691551 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:36:15.691385    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2ec6829-400f-42cc-bb98-f1f084ce16f9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6\" (UID: \"c2ec6829-400f-42cc-bb98-f1f084ce16f9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6"
Sep 09 15:36:15.691551 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:36:15.691417    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2ec6829-400f-42cc-bb98-f1f084ce16f9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6\" (UID: \"c2ec6829-400f-42cc-bb98-f1f084ce16f9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6"
Sep 09 15:36:15.691551 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:36:15.691480    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdf4j\" (UniqueName: \"kubernetes.io/projected/c2ec6829-400f-42cc-bb98-f1f084ce16f9-kube-api-access-hdf4j\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6\" (UID: \"c2ec6829-400f-42cc-bb98-f1f084ce16f9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6"
Sep 09 15:36:15.691551 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:36:15.691539    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2ec6829-400f-42cc-bb98-f1f084ce16f9-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6\" (UID: \"c2ec6829-400f-42cc-bb98-f1f084ce16f9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6"
Sep 09 15:36:15.691680 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:36:15.691564    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2ec6829-400f-42cc-bb98-f1f084ce16f9-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6\" (UID: \"c2ec6829-400f-42cc-bb98-f1f084ce16f9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6"
Sep 09 15:36:15.792606 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:36:15.792564    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2ec6829-400f-42cc-bb98-f1f084ce16f9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6\" (UID: \"c2ec6829-400f-42cc-bb98-f1f084ce16f9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6"
Sep 09 15:36:15.792821 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:36:15.792618    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdf4j\" (UniqueName: \"kubernetes.io/projected/c2ec6829-400f-42cc-bb98-f1f084ce16f9-kube-api-access-hdf4j\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6\" (UID: \"c2ec6829-400f-42cc-bb98-f1f084ce16f9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6"
Sep 09 15:36:15.792821 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:36:15.792657    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2ec6829-400f-42cc-bb98-f1f084ce16f9-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6\" (UID: \"c2ec6829-400f-42cc-bb98-f1f084ce16f9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6"
Sep 09 15:36:15.792821 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:36:15.792677    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2ec6829-400f-42cc-bb98-f1f084ce16f9-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6\" (UID: \"c2ec6829-400f-42cc-bb98-f1f084ce16f9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6"
Sep 09 15:36:15.792821 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:36:15.792704    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2ec6829-400f-42cc-bb98-f1f084ce16f9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6\" (UID: \"c2ec6829-400f-42cc-bb98-f1f084ce16f9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6"
Sep 09 15:36:15.792821 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:36:15.792770    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2ec6829-400f-42cc-bb98-f1f084ce16f9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6\" (UID: \"c2ec6829-400f-42cc-bb98-f1f084ce16f9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6"
Sep 09 15:36:15.793235 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:36:15.793211    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2ec6829-400f-42cc-bb98-f1f084ce16f9-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6\" (UID: \"c2ec6829-400f-42cc-bb98-f1f084ce16f9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6"
Sep 09 15:36:15.793311 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:36:15.793238    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2ec6829-400f-42cc-bb98-f1f084ce16f9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6\" (UID: \"c2ec6829-400f-42cc-bb98-f1f084ce16f9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6"
Sep 09 15:36:15.793311 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:36:15.793282    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2ec6829-400f-42cc-bb98-f1f084ce16f9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6\" (UID: \"c2ec6829-400f-42cc-bb98-f1f084ce16f9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6"
Sep 09 15:36:15.795091 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:36:15.795061    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2ec6829-400f-42cc-bb98-f1f084ce16f9-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6\" (UID: \"c2ec6829-400f-42cc-bb98-f1f084ce16f9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6"
Sep 09 15:36:15.795336 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:36:15.795319    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2ec6829-400f-42cc-bb98-f1f084ce16f9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6\" (UID: \"c2ec6829-400f-42cc-bb98-f1f084ce16f9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6"
Sep 09 15:36:15.803204 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:36:15.803175    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdf4j\" (UniqueName: \"kubernetes.io/projected/c2ec6829-400f-42cc-bb98-f1f084ce16f9-kube-api-access-hdf4j\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6\" (UID: \"c2ec6829-400f-42cc-bb98-f1f084ce16f9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6"
Sep 09 15:36:15.873689 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:36:15.873602    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6"
Sep 09 15:36:16.144478 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:36:16.144309    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6"]
Sep 09 15:36:16.147224 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:36:16.147195    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2ec6829_400f_42cc_bb98_f1f084ce16f9.slice/crio-c24c23ead0eb758a85d6ef0925f994e1227b5150767b002ac744761f14956598 WatchSource:0}: Error finding container c24c23ead0eb758a85d6ef0925f994e1227b5150767b002ac744761f14956598: Status 404 returned error can't find the container with id c24c23ead0eb758a85d6ef0925f994e1227b5150767b002ac744761f14956598
Sep 09 15:36:16.400797 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:36:16.400764    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6" event={"ID":"c2ec6829-400f-42cc-bb98-f1f084ce16f9","Type":"ContainerStarted","Data":"c24c23ead0eb758a85d6ef0925f994e1227b5150767b002ac744761f14956598"}
Sep 09 15:36:17.406484 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:36:17.406450    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6" event={"ID":"c2ec6829-400f-42cc-bb98-f1f084ce16f9","Type":"ContainerStarted","Data":"06b456a32e11ad04eaf399deca7046b11b15f458335b77b79a11f770b5476382"}
Sep 09 15:36:17.424229 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:36:17.424170    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6" podStartSLOduration=2.1809860150000002 podStartE2EDuration="2.424154133s" totalImagesPullingTime="243.168118ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:36:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:36:17.423241838 +0000 UTC m=+943.022136481" watchObservedRunningTime="2026-09-09 15:36:17.424154133 +0000 UTC m=+943.023048776"
Sep 09 15:36:25.874471 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:36:25.874418    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6"
Sep 09 15:36:25.874903 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:36:25.874490    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6"
Sep 09 15:36:25.887250 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:36:25.887224    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6"
Sep 09 15:36:26.454907 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:36:26.454878    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6"
Sep 09 15:37:23.531191 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:23.531105    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d7cdcc7d7-vdtg8"]
Sep 09 15:37:23.534409 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:23.534392    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d7cdcc7d7-vdtg8"
Sep 09 15:37:23.538341 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:23.538317    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9frgw\""
Sep 09 15:37:23.538341 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:23.538327    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 09 15:37:23.542553 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:23.542406    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d7cdcc7d7-vdtg8"]
Sep 09 15:37:23.595686 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:23.595647    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8f94a3e7-a91a-48fc-8f92-0a2e9eaa999e-tls-cert\") pod \"authorino-5d7cdcc7d7-vdtg8\" (UID: \"8f94a3e7-a91a-48fc-8f92-0a2e9eaa999e\") " pod="kuadrant-system/authorino-5d7cdcc7d7-vdtg8"
Sep 09 15:37:23.595686 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:23.595688    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztb99\" (UniqueName: \"kubernetes.io/projected/8f94a3e7-a91a-48fc-8f92-0a2e9eaa999e-kube-api-access-ztb99\") pod \"authorino-5d7cdcc7d7-vdtg8\" (UID: \"8f94a3e7-a91a-48fc-8f92-0a2e9eaa999e\") " pod="kuadrant-system/authorino-5d7cdcc7d7-vdtg8"
Sep 09 15:37:23.696526 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:23.696493    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8f94a3e7-a91a-48fc-8f92-0a2e9eaa999e-tls-cert\") pod \"authorino-5d7cdcc7d7-vdtg8\" (UID: \"8f94a3e7-a91a-48fc-8f92-0a2e9eaa999e\") " pod="kuadrant-system/authorino-5d7cdcc7d7-vdtg8"
Sep 09 15:37:23.696526 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:23.696527    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztb99\" (UniqueName: \"kubernetes.io/projected/8f94a3e7-a91a-48fc-8f92-0a2e9eaa999e-kube-api-access-ztb99\") pod \"authorino-5d7cdcc7d7-vdtg8\" (UID: \"8f94a3e7-a91a-48fc-8f92-0a2e9eaa999e\") " pod="kuadrant-system/authorino-5d7cdcc7d7-vdtg8"
Sep 09 15:37:23.699035 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:23.699007    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8f94a3e7-a91a-48fc-8f92-0a2e9eaa999e-tls-cert\") pod \"authorino-5d7cdcc7d7-vdtg8\" (UID: \"8f94a3e7-a91a-48fc-8f92-0a2e9eaa999e\") " pod="kuadrant-system/authorino-5d7cdcc7d7-vdtg8"
Sep 09 15:37:23.707039 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:23.707015    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztb99\" (UniqueName: \"kubernetes.io/projected/8f94a3e7-a91a-48fc-8f92-0a2e9eaa999e-kube-api-access-ztb99\") pod \"authorino-5d7cdcc7d7-vdtg8\" (UID: \"8f94a3e7-a91a-48fc-8f92-0a2e9eaa999e\") " pod="kuadrant-system/authorino-5d7cdcc7d7-vdtg8"
Sep 09 15:37:23.845185 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:23.845088    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d7cdcc7d7-vdtg8"
Sep 09 15:37:24.114194 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:24.114122    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d7cdcc7d7-vdtg8"]
Sep 09 15:37:24.117565 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:37:24.117531    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f94a3e7_a91a_48fc_8f92_0a2e9eaa999e.slice/crio-626d7fafa0d1df5da58f463a8031454bc25596930b377cfefda4482c1d4fcf09 WatchSource:0}: Error finding container 626d7fafa0d1df5da58f463a8031454bc25596930b377cfefda4482c1d4fcf09: Status 404 returned error can't find the container with id 626d7fafa0d1df5da58f463a8031454bc25596930b377cfefda4482c1d4fcf09
Sep 09 15:37:24.702825 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:24.702780    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d7cdcc7d7-vdtg8" event={"ID":"8f94a3e7-a91a-48fc-8f92-0a2e9eaa999e","Type":"ContainerStarted","Data":"626d7fafa0d1df5da58f463a8031454bc25596930b377cfefda4482c1d4fcf09"}
Sep 09 15:37:28.720395 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:28.720361    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d7cdcc7d7-vdtg8" event={"ID":"8f94a3e7-a91a-48fc-8f92-0a2e9eaa999e","Type":"ContainerStarted","Data":"e8faf2cb6da23f1e42024a62d5a362c4cd9374deccd864f340b5c389bae85095"}
Sep 09 15:37:28.737638 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:28.737581    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d7cdcc7d7-vdtg8" podStartSLOduration=1.251943783 podStartE2EDuration="5.737566131s" totalImagesPullingTime="4.485622348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:37:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:37:28.736308854 +0000 UTC m=+1014.335203497" watchObservedRunningTime="2026-09-09 15:37:28.737566131 +0000 UTC m=+1014.336460775"
Sep 09 15:37:48.657760 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.657709    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"]
Sep 09 15:37:48.661071 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.661048    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"
Sep 09 15:37:48.663589 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.663566    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-df30be-openshift-default-dockercfg-zlhf6\""
Sep 09 15:37:48.675702 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.675676    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"]
Sep 09 15:37:48.714416 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.714366    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0eec3f-27bc-4191-a148-4c5a8a531611-istio-envoy\") pod \"e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k\" (UID: \"3b0eec3f-27bc-4191-a148-4c5a8a531611\") " pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"
Sep 09 15:37:48.714591 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.714424    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0eec3f-27bc-4191-a148-4c5a8a531611-istio-podinfo\") pod \"e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k\" (UID: \"3b0eec3f-27bc-4191-a148-4c5a8a531611\") " pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"
Sep 09 15:37:48.714591 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.714503    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p278h\" (UniqueName: \"kubernetes.io/projected/3b0eec3f-27bc-4191-a148-4c5a8a531611-kube-api-access-p278h\") pod \"e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k\" (UID: \"3b0eec3f-27bc-4191-a148-4c5a8a531611\") " pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"
Sep 09 15:37:48.714591 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.714565    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0eec3f-27bc-4191-a148-4c5a8a531611-istio-data\") pod \"e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k\" (UID: \"3b0eec3f-27bc-4191-a148-4c5a8a531611\") " pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"
Sep 09 15:37:48.714840 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.714611    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0eec3f-27bc-4191-a148-4c5a8a531611-credential-socket\") pod \"e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k\" (UID: \"3b0eec3f-27bc-4191-a148-4c5a8a531611\") " pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"
Sep 09 15:37:48.714840 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.714643    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0eec3f-27bc-4191-a148-4c5a8a531611-istiod-ca-cert\") pod \"e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k\" (UID: \"3b0eec3f-27bc-4191-a148-4c5a8a531611\") " pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"
Sep 09 15:37:48.714840 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.714703    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0eec3f-27bc-4191-a148-4c5a8a531611-workload-certs\") pod \"e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k\" (UID: \"3b0eec3f-27bc-4191-a148-4c5a8a531611\") " pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"
Sep 09 15:37:48.714840 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.714769    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b0eec3f-27bc-4191-a148-4c5a8a531611-istio-token\") pod \"e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k\" (UID: \"3b0eec3f-27bc-4191-a148-4c5a8a531611\") " pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"
Sep 09 15:37:48.714840 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.714797    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0eec3f-27bc-4191-a148-4c5a8a531611-workload-socket\") pod \"e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k\" (UID: \"3b0eec3f-27bc-4191-a148-4c5a8a531611\") " pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"
Sep 09 15:37:48.809946 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.809910    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"]
Sep 09 15:37:48.813655 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.813623    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"
Sep 09 15:37:48.815266 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.815243    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0eec3f-27bc-4191-a148-4c5a8a531611-workload-certs\") pod \"e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k\" (UID: \"3b0eec3f-27bc-4191-a148-4c5a8a531611\") " pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"
Sep 09 15:37:48.815394 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.815293    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b0eec3f-27bc-4191-a148-4c5a8a531611-istio-token\") pod \"e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k\" (UID: \"3b0eec3f-27bc-4191-a148-4c5a8a531611\") " pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"
Sep 09 15:37:48.815394 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.815321    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0eec3f-27bc-4191-a148-4c5a8a531611-workload-socket\") pod \"e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k\" (UID: \"3b0eec3f-27bc-4191-a148-4c5a8a531611\") " pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"
Sep 09 15:37:48.815504 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.815397    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0eec3f-27bc-4191-a148-4c5a8a531611-istio-envoy\") pod \"e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k\" (UID: \"3b0eec3f-27bc-4191-a148-4c5a8a531611\") " pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"
Sep 09 15:37:48.815504 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.815429    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0eec3f-27bc-4191-a148-4c5a8a531611-istio-podinfo\") pod \"e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k\" (UID: \"3b0eec3f-27bc-4191-a148-4c5a8a531611\") " pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"
Sep 09 15:37:48.815504 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.815461    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p278h\" (UniqueName: \"kubernetes.io/projected/3b0eec3f-27bc-4191-a148-4c5a8a531611-kube-api-access-p278h\") pod \"e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k\" (UID: \"3b0eec3f-27bc-4191-a148-4c5a8a531611\") " pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"
Sep 09 15:37:48.815642 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.815501    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0eec3f-27bc-4191-a148-4c5a8a531611-istio-data\") pod \"e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k\" (UID: \"3b0eec3f-27bc-4191-a148-4c5a8a531611\") " pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"
Sep 09 15:37:48.815642 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.815531    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0eec3f-27bc-4191-a148-4c5a8a531611-credential-socket\") pod \"e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k\" (UID: \"3b0eec3f-27bc-4191-a148-4c5a8a531611\") " pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"
Sep 09 15:37:48.815642 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.815563    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0eec3f-27bc-4191-a148-4c5a8a531611-istiod-ca-cert\") pod \"e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k\" (UID: \"3b0eec3f-27bc-4191-a148-4c5a8a531611\") " pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"
Sep 09 15:37:48.816389 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.816360    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0eec3f-27bc-4191-a148-4c5a8a531611-credential-socket\") pod \"e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k\" (UID: \"3b0eec3f-27bc-4191-a148-4c5a8a531611\") " pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"
Sep 09 15:37:48.816488 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.816448    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0eec3f-27bc-4191-a148-4c5a8a531611-istiod-ca-cert\") pod \"e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k\" (UID: \"3b0eec3f-27bc-4191-a148-4c5a8a531611\") " pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"
Sep 09 15:37:48.816668 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.816649    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0eec3f-27bc-4191-a148-4c5a8a531611-workload-certs\") pod \"e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k\" (UID: \"3b0eec3f-27bc-4191-a148-4c5a8a531611\") " pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"
Sep 09 15:37:48.817287 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.816668    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-1e15c8-openshift-default-dockercfg-4s498\""
Sep 09 15:37:48.817559 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.816901    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0eec3f-27bc-4191-a148-4c5a8a531611-workload-socket\") pod \"e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k\" (UID: \"3b0eec3f-27bc-4191-a148-4c5a8a531611\") " pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"
Sep 09 15:37:48.817713 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.817165    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0eec3f-27bc-4191-a148-4c5a8a531611-istio-data\") pod \"e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k\" (UID: \"3b0eec3f-27bc-4191-a148-4c5a8a531611\") " pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"
Sep 09 15:37:48.820045 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.819997    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0eec3f-27bc-4191-a148-4c5a8a531611-istio-envoy\") pod \"e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k\" (UID: \"3b0eec3f-27bc-4191-a148-4c5a8a531611\") " pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"
Sep 09 15:37:48.820259 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.820239    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0eec3f-27bc-4191-a148-4c5a8a531611-istio-podinfo\") pod \"e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k\" (UID: \"3b0eec3f-27bc-4191-a148-4c5a8a531611\") " pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"
Sep 09 15:37:48.825797 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.825719    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"]
Sep 09 15:37:48.833933 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.833908    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p278h\" (UniqueName: \"kubernetes.io/projected/3b0eec3f-27bc-4191-a148-4c5a8a531611-kube-api-access-p278h\") pod \"e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k\" (UID: \"3b0eec3f-27bc-4191-a148-4c5a8a531611\") " pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"
Sep 09 15:37:48.835803 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.835770    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b0eec3f-27bc-4191-a148-4c5a8a531611-istio-token\") pod \"e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k\" (UID: \"3b0eec3f-27bc-4191-a148-4c5a8a531611\") " pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"
Sep 09 15:37:48.916831 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.916714    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6xc5\" (UniqueName: \"kubernetes.io/projected/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-kube-api-access-s6xc5\") pod \"e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv\" (UID: \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\") " pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"
Sep 09 15:37:48.916831 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.916802    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-istio-data\") pod \"e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv\" (UID: \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\") " pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"
Sep 09 15:37:48.917054 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.916833    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-istio-token\") pod \"e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv\" (UID: \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\") " pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"
Sep 09 15:37:48.917054 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.916864    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-istio-podinfo\") pod \"e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv\" (UID: \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\") " pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"
Sep 09 15:37:48.917054 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.916953    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-credential-socket\") pod \"e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv\" (UID: \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\") " pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"
Sep 09 15:37:48.917054 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.917030    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-istio-envoy\") pod \"e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv\" (UID: \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\") " pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"
Sep 09 15:37:48.917262 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.917063    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-workload-socket\") pod \"e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv\" (UID: \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\") " pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"
Sep 09 15:37:48.917262 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.917103    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-istiod-ca-cert\") pod \"e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv\" (UID: \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\") " pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"
Sep 09 15:37:48.917262 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.917147    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-workload-certs\") pod \"e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv\" (UID: \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\") " pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"
Sep 09 15:37:48.971407 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:48.971374    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"
Sep 09 15:37:49.017935 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:49.017884    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-istio-podinfo\") pod \"e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv\" (UID: \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\") " pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"
Sep 09 15:37:49.018124 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:49.017978    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-credential-socket\") pod \"e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv\" (UID: \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\") " pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"
Sep 09 15:37:49.018124 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:49.018076    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-istio-envoy\") pod \"e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv\" (UID: \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\") " pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"
Sep 09 15:37:49.018124 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:49.018107    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-workload-socket\") pod \"e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv\" (UID: \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\") " pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"
Sep 09 15:37:49.018285 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:49.018150    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-istiod-ca-cert\") pod \"e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv\" (UID: \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\") " pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"
Sep 09 15:37:49.018285 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:49.018181    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-workload-certs\") pod \"e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv\" (UID: \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\") " pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"
Sep 09 15:37:49.018285 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:49.018231    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6xc5\" (UniqueName: \"kubernetes.io/projected/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-kube-api-access-s6xc5\") pod \"e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv\" (UID: \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\") " pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"
Sep 09 15:37:49.018433 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:49.018297    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-istio-data\") pod \"e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv\" (UID: \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\") " pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"
Sep 09 15:37:49.018433 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:49.018331    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-istio-token\") pod \"e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv\" (UID: \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\") " pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"
Sep 09 15:37:49.019602 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:49.018703    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-workload-socket\") pod \"e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv\" (UID: \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\") " pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"
Sep 09 15:37:49.019602 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:49.019221    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-workload-certs\") pod \"e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv\" (UID: \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\") " pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"
Sep 09 15:37:49.019602 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:49.019325    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-istio-data\") pod \"e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv\" (UID: \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\") " pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"
Sep 09 15:37:49.019602 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:49.019537    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-istiod-ca-cert\") pod \"e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv\" (UID: \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\") " pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"
Sep 09 15:37:49.019977 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:49.019939    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-credential-socket\") pod \"e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv\" (UID: \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\") " pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"
Sep 09 15:37:49.021909 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:49.021881    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-istio-podinfo\") pod \"e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv\" (UID: \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\") " pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"
Sep 09 15:37:49.022408 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:49.022383    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-istio-envoy\") pod \"e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv\" (UID: \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\") " pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"
Sep 09 15:37:49.063345 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:49.063268    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6xc5\" (UniqueName: \"kubernetes.io/projected/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-kube-api-access-s6xc5\") pod \"e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv\" (UID: \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\") " pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"
Sep 09 15:37:49.063951 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:49.063906    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-istio-token\") pod \"e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv\" (UID: \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\") " pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"
Sep 09 15:37:49.154070 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:49.154035    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"
Sep 09 15:37:49.267703 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:49.267668    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"]
Sep 09 15:37:49.270594 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:37:49.270564    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b0eec3f_27bc_4191_a148_4c5a8a531611.slice/crio-8b7d6776a47e7dfd87c1125c16fa23c167f9d95cd551bb5339312375d9c5726f WatchSource:0}: Error finding container 8b7d6776a47e7dfd87c1125c16fa23c167f9d95cd551bb5339312375d9c5726f: Status 404 returned error can't find the container with id 8b7d6776a47e7dfd87c1125c16fa23c167f9d95cd551bb5339312375d9c5726f
Sep 09 15:37:49.272978 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:49.272941    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:37:49.273075 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:49.273030    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:37:49.273134 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:49.273084    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:37:49.437574 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:49.437540    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"]
Sep 09 15:37:49.441201 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:37:49.441169    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5778704d_c2e2_4553_b8a5_0a7ebf84f9dc.slice/crio-b0b5c688698a30843e1eda71e4a13f6cf7407bdcdefcaa3b56daee6951ba9f24 WatchSource:0}: Error finding container b0b5c688698a30843e1eda71e4a13f6cf7407bdcdefcaa3b56daee6951ba9f24: Status 404 returned error can't find the container with id b0b5c688698a30843e1eda71e4a13f6cf7407bdcdefcaa3b56daee6951ba9f24
Sep 09 15:37:49.444064 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:49.444029    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:37:49.444154 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:49.444108    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:37:49.444213 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:49.444160    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:37:49.818701 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:49.818610    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv" event={"ID":"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc","Type":"ContainerStarted","Data":"b8833e8ec397320bddc2396cb9a39708c97e818883a51aea284024664971fa8b"}
Sep 09 15:37:49.819163 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:49.818716    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv" event={"ID":"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc","Type":"ContainerStarted","Data":"b0b5c688698a30843e1eda71e4a13f6cf7407bdcdefcaa3b56daee6951ba9f24"}
Sep 09 15:37:49.820266 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:49.820238    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k" event={"ID":"3b0eec3f-27bc-4191-a148-4c5a8a531611","Type":"ContainerStarted","Data":"635dd9ae07ff8cfe25f6de4ddff68a105eab182d691bd46247c98ed82d369912"}
Sep 09 15:37:49.820385 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:49.820293    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k" event={"ID":"3b0eec3f-27bc-4191-a148-4c5a8a531611","Type":"ContainerStarted","Data":"8b7d6776a47e7dfd87c1125c16fa23c167f9d95cd551bb5339312375d9c5726f"}
Sep 09 15:37:49.845423 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:49.845355    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv" podStartSLOduration=1.845335046 podStartE2EDuration="1.845335046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:37:49.845049403 +0000 UTC m=+1035.443944085" watchObservedRunningTime="2026-09-09 15:37:49.845335046 +0000 UTC m=+1035.444229691"
Sep 09 15:37:49.866337 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:49.866279    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k" podStartSLOduration=1.8662620410000001 podStartE2EDuration="1.866262041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:37:49.865848989 +0000 UTC m=+1035.464743645" watchObservedRunningTime="2026-09-09 15:37:49.866262041 +0000 UTC m=+1035.465156684"
Sep 09 15:37:49.972177 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:49.972139    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"
Sep 09 15:37:50.154894 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:50.154802    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"
Sep 09 15:37:50.160017 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:50.159988    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"
Sep 09 15:37:50.825014 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:50.824983    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"
Sep 09 15:37:50.825869 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:50.825844    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"
Sep 09 15:37:50.977209 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:50.977173    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"
Sep 09 15:37:51.829628 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:51.829592    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"
Sep 09 15:37:51.830638 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:51.830620    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"
Sep 09 15:37:53.862180 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:53.862138    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-df30be-57ff546d6b-ts9mj"]
Sep 09 15:37:53.865656 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:53.865634    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-df30be-57ff546d6b-ts9mj"
Sep 09 15:37:53.868430 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:53.868409    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-df30be\""
Sep 09 15:37:53.868783 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:53.868768    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-df30be-dockercfg-jq7nd\""
Sep 09 15:37:53.878098 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:53.878068    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-df30be-57ff546d6b-ts9mj"]
Sep 09 15:37:53.966660 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:53.966621    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxfkl\" (UniqueName: \"kubernetes.io/projected/19eb8f8e-5347-4a51-8b90-cdbf2b1102f9-kube-api-access-qxfkl\") pod \"payload-pre-processing-e2e-worker-w0-df30be-57ff546d6b-ts9mj\" (UID: \"19eb8f8e-5347-4a51-8b90-cdbf2b1102f9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-df30be-57ff546d6b-ts9mj"
Sep 09 15:37:53.966867 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:53.966668    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19eb8f8e-5347-4a51-8b90-cdbf2b1102f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-df30be-57ff546d6b-ts9mj\" (UID: \"19eb8f8e-5347-4a51-8b90-cdbf2b1102f9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-df30be-57ff546d6b-ts9mj"
Sep 09 15:37:54.068075 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:54.068021    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxfkl\" (UniqueName: \"kubernetes.io/projected/19eb8f8e-5347-4a51-8b90-cdbf2b1102f9-kube-api-access-qxfkl\") pod \"payload-pre-processing-e2e-worker-w0-df30be-57ff546d6b-ts9mj\" (UID: \"19eb8f8e-5347-4a51-8b90-cdbf2b1102f9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-df30be-57ff546d6b-ts9mj"
Sep 09 15:37:54.068249 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:54.068091    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19eb8f8e-5347-4a51-8b90-cdbf2b1102f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-df30be-57ff546d6b-ts9mj\" (UID: \"19eb8f8e-5347-4a51-8b90-cdbf2b1102f9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-df30be-57ff546d6b-ts9mj"
Sep 09 15:37:54.068850 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:54.068825    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19eb8f8e-5347-4a51-8b90-cdbf2b1102f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-df30be-57ff546d6b-ts9mj\" (UID: \"19eb8f8e-5347-4a51-8b90-cdbf2b1102f9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-df30be-57ff546d6b-ts9mj"
Sep 09 15:37:54.075878 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:54.075856    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxfkl\" (UniqueName: \"kubernetes.io/projected/19eb8f8e-5347-4a51-8b90-cdbf2b1102f9-kube-api-access-qxfkl\") pod \"payload-pre-processing-e2e-worker-w0-df30be-57ff546d6b-ts9mj\" (UID: \"19eb8f8e-5347-4a51-8b90-cdbf2b1102f9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-df30be-57ff546d6b-ts9mj"
Sep 09 15:37:54.176454 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:54.176417    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-df30be-57ff546d6b-ts9mj"
Sep 09 15:37:54.445788 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:54.445683    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-df30be-57ff546d6b-ts9mj"]
Sep 09 15:37:54.449149 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:37:54.449117    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19eb8f8e_5347_4a51_8b90_cdbf2b1102f9.slice/crio-68ec637209149ea8457fefc94377ec7df44b5809a3d8beb04be63881dccfc64a WatchSource:0}: Error finding container 68ec637209149ea8457fefc94377ec7df44b5809a3d8beb04be63881dccfc64a: Status 404 returned error can't find the container with id 68ec637209149ea8457fefc94377ec7df44b5809a3d8beb04be63881dccfc64a
Sep 09 15:37:54.846296 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:54.846184    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-df30be-57ff546d6b-ts9mj" event={"ID":"19eb8f8e-5347-4a51-8b90-cdbf2b1102f9","Type":"ContainerStarted","Data":"7e2d6ee1eea4877a6c786d237fc6c515b9cbaaa67fa58ddaf6f7ddb06a8508fa"}
Sep 09 15:37:54.846296 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:54.846252    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-df30be-57ff546d6b-ts9mj" event={"ID":"19eb8f8e-5347-4a51-8b90-cdbf2b1102f9","Type":"ContainerStarted","Data":"68ec637209149ea8457fefc94377ec7df44b5809a3d8beb04be63881dccfc64a"}
Sep 09 15:37:54.846612 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:54.846384    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-df30be-57ff546d6b-ts9mj"
Sep 09 15:37:54.862094 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:54.862029    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-df30be-57ff546d6b-ts9mj" podStartSLOduration=1.862005039 podStartE2EDuration="1.862005039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:37:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:37:54.861674304 +0000 UTC m=+1040.460568949" watchObservedRunningTime="2026-09-09 15:37:54.862005039 +0000 UTC m=+1040.460899684"
Sep 09 15:37:54.945553 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:54.945514    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-1e15c8-6f968d8478-8gkmw"]
Sep 09 15:37:54.949416 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:54.949393    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1e15c8-6f968d8478-8gkmw"
Sep 09 15:37:54.952003 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:54.951957    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-1e15c8\""
Sep 09 15:37:54.966035 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:54.966007    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-1e15c8-6f968d8478-8gkmw"]
Sep 09 15:37:54.987310 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:54.987275    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1e15c8-7c96479557-8f8jr"]
Sep 09 15:37:54.990509 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:54.990494    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-1e15c8-7c96479557-8f8jr"
Sep 09 15:37:54.994009 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:54.993991    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-1e15c8-dockercfg-7lm8l\""
Sep 09 15:37:55.001377 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:55.001353    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1e15c8-7c96479557-8f8jr"]
Sep 09 15:37:55.078767 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:55.078712    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39cb9b9a-9e10-4879-ad05-99155842ca9e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-1e15c8-7c96479557-8f8jr\" (UID: \"39cb9b9a-9e10-4879-ad05-99155842ca9e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1e15c8-7c96479557-8f8jr"
Sep 09 15:37:55.078963 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:55.078791    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cmf4\" (UniqueName: \"kubernetes.io/projected/39cb9b9a-9e10-4879-ad05-99155842ca9e-kube-api-access-9cmf4\") pod \"payload-processing-e2e-worker-w4-1e15c8-7c96479557-8f8jr\" (UID: \"39cb9b9a-9e10-4879-ad05-99155842ca9e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1e15c8-7c96479557-8f8jr"
Sep 09 15:37:55.078963 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:55.078891    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1152d276-323d-4e1a-b67e-a2ce9927ae59-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-1e15c8-6f968d8478-8gkmw\" (UID: \"1152d276-323d-4e1a-b67e-a2ce9927ae59\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1e15c8-6f968d8478-8gkmw"
Sep 09 15:37:55.079090 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:55.078981    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mg6t\" (UniqueName: \"kubernetes.io/projected/1152d276-323d-4e1a-b67e-a2ce9927ae59-kube-api-access-8mg6t\") pod \"payload-pre-processing-e2e-worker-w4-1e15c8-6f968d8478-8gkmw\" (UID: \"1152d276-323d-4e1a-b67e-a2ce9927ae59\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1e15c8-6f968d8478-8gkmw"
Sep 09 15:37:55.179801 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:55.179766    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mg6t\" (UniqueName: \"kubernetes.io/projected/1152d276-323d-4e1a-b67e-a2ce9927ae59-kube-api-access-8mg6t\") pod \"payload-pre-processing-e2e-worker-w4-1e15c8-6f968d8478-8gkmw\" (UID: \"1152d276-323d-4e1a-b67e-a2ce9927ae59\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1e15c8-6f968d8478-8gkmw"
Sep 09 15:37:55.179982 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:55.179846    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39cb9b9a-9e10-4879-ad05-99155842ca9e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-1e15c8-7c96479557-8f8jr\" (UID: \"39cb9b9a-9e10-4879-ad05-99155842ca9e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1e15c8-7c96479557-8f8jr"
Sep 09 15:37:55.179982 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:55.179894    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cmf4\" (UniqueName: \"kubernetes.io/projected/39cb9b9a-9e10-4879-ad05-99155842ca9e-kube-api-access-9cmf4\") pod \"payload-processing-e2e-worker-w4-1e15c8-7c96479557-8f8jr\" (UID: \"39cb9b9a-9e10-4879-ad05-99155842ca9e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1e15c8-7c96479557-8f8jr"
Sep 09 15:37:55.179982 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:55.179960    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1152d276-323d-4e1a-b67e-a2ce9927ae59-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-1e15c8-6f968d8478-8gkmw\" (UID: \"1152d276-323d-4e1a-b67e-a2ce9927ae59\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1e15c8-6f968d8478-8gkmw"
Sep 09 15:37:55.180601 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:55.180578    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39cb9b9a-9e10-4879-ad05-99155842ca9e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-1e15c8-7c96479557-8f8jr\" (UID: \"39cb9b9a-9e10-4879-ad05-99155842ca9e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1e15c8-7c96479557-8f8jr"
Sep 09 15:37:55.180694 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:55.180602    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1152d276-323d-4e1a-b67e-a2ce9927ae59-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-1e15c8-6f968d8478-8gkmw\" (UID: \"1152d276-323d-4e1a-b67e-a2ce9927ae59\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1e15c8-6f968d8478-8gkmw"
Sep 09 15:37:55.187982 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:55.187952    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mg6t\" (UniqueName: \"kubernetes.io/projected/1152d276-323d-4e1a-b67e-a2ce9927ae59-kube-api-access-8mg6t\") pod \"payload-pre-processing-e2e-worker-w4-1e15c8-6f968d8478-8gkmw\" (UID: \"1152d276-323d-4e1a-b67e-a2ce9927ae59\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1e15c8-6f968d8478-8gkmw"
Sep 09 15:37:55.188090 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:55.187996    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cmf4\" (UniqueName: \"kubernetes.io/projected/39cb9b9a-9e10-4879-ad05-99155842ca9e-kube-api-access-9cmf4\") pod \"payload-processing-e2e-worker-w4-1e15c8-7c96479557-8f8jr\" (UID: \"39cb9b9a-9e10-4879-ad05-99155842ca9e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1e15c8-7c96479557-8f8jr"
Sep 09 15:37:55.264171 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:55.264135    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1e15c8-6f968d8478-8gkmw"
Sep 09 15:37:55.300317 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:55.300281    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-1e15c8-7c96479557-8f8jr"
Sep 09 15:37:55.551871 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:55.551841    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-1e15c8-6f968d8478-8gkmw"]
Sep 09 15:37:55.554933 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:37:55.554901    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1152d276_323d_4e1a_b67e_a2ce9927ae59.slice/crio-8c602ed72f1505cac152aab82386f954840021f61ee7e8ecefb7275404f8b74c WatchSource:0}: Error finding container 8c602ed72f1505cac152aab82386f954840021f61ee7e8ecefb7275404f8b74c: Status 404 returned error can't find the container with id 8c602ed72f1505cac152aab82386f954840021f61ee7e8ecefb7275404f8b74c
Sep 09 15:37:55.585877 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:55.585791    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1e15c8-7c96479557-8f8jr"]
Sep 09 15:37:55.591836 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:37:55.591811    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39cb9b9a_9e10_4879_ad05_99155842ca9e.slice/crio-a64842e58c1b7f1b87edbe775adba4c26ccc6387d9c44f49b129c04a61e6877c WatchSource:0}: Error finding container a64842e58c1b7f1b87edbe775adba4c26ccc6387d9c44f49b129c04a61e6877c: Status 404 returned error can't find the container with id a64842e58c1b7f1b87edbe775adba4c26ccc6387d9c44f49b129c04a61e6877c
Sep 09 15:37:55.853720 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:55.853624    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1e15c8-6f968d8478-8gkmw" event={"ID":"1152d276-323d-4e1a-b67e-a2ce9927ae59","Type":"ContainerStarted","Data":"53f4cec957b58b7e641fb97ed0a3ea04a86addd43dbdd026088bd4ccca26ea75"}
Sep 09 15:37:55.853720 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:55.853685    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1e15c8-6f968d8478-8gkmw" event={"ID":"1152d276-323d-4e1a-b67e-a2ce9927ae59","Type":"ContainerStarted","Data":"8c602ed72f1505cac152aab82386f954840021f61ee7e8ecefb7275404f8b74c"}
Sep 09 15:37:55.853957 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:55.853726    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1e15c8-6f968d8478-8gkmw"
Sep 09 15:37:55.855795 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:55.855763    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-1e15c8-7c96479557-8f8jr" event={"ID":"39cb9b9a-9e10-4879-ad05-99155842ca9e","Type":"ContainerStarted","Data":"13f1abea79589731d7be57557aac915377b3ce68926e705d0583d2bbb049fe6b"}
Sep 09 15:37:55.855910 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:55.855818    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-1e15c8-7c96479557-8f8jr" event={"ID":"39cb9b9a-9e10-4879-ad05-99155842ca9e","Type":"ContainerStarted","Data":"a64842e58c1b7f1b87edbe775adba4c26ccc6387d9c44f49b129c04a61e6877c"}
Sep 09 15:37:55.855976 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:55.855939    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-1e15c8-7c96479557-8f8jr"
Sep 09 15:37:55.872100 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:55.872035    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1e15c8-6f968d8478-8gkmw" podStartSLOduration=1.8720130400000001 podStartE2EDuration="1.87201304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:37:55.869843621 +0000 UTC m=+1041.468738265" watchObservedRunningTime="2026-09-09 15:37:55.87201304 +0000 UTC m=+1041.470907690"
Sep 09 15:37:55.875703 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:55.875674    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-57b579-58954c5b5d-dvsnc"]
Sep 09 15:37:55.879392 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:55.879364    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-57b579-58954c5b5d-dvsnc"
Sep 09 15:37:55.881873 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:55.881851    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-57b579\""
Sep 09 15:37:55.881996 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:55.881850    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-57b579-dockercfg-rh5h7\""
Sep 09 15:37:55.889442 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:55.889384    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-57b579-58954c5b5d-dvsnc"]
Sep 09 15:37:55.890174 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:55.890116    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-1e15c8-7c96479557-8f8jr" podStartSLOduration=1.890098262 podStartE2EDuration="1.890098262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:37:55.889726156 +0000 UTC m=+1041.488620806" watchObservedRunningTime="2026-09-09 15:37:55.890098262 +0000 UTC m=+1041.488992910"
Sep 09 15:37:55.903688 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:55.903660    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-57b579-7994b7cc87-fgscr"]
Sep 09 15:37:55.907370 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:55.907350    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-57b579-7994b7cc87-fgscr"
Sep 09 15:37:55.915214 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:55.915185    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-57b579-7994b7cc87-fgscr"]
Sep 09 15:37:55.987577 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:55.987537    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/015c0cd8-414e-4d52-aeee-c64c1588b315-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-57b579-58954c5b5d-dvsnc\" (UID: \"015c0cd8-414e-4d52-aeee-c64c1588b315\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-57b579-58954c5b5d-dvsnc"
Sep 09 15:37:55.987577 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:55.987577    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d33f110-3b40-4098-8075-2a42dba9c751-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-57b579-7994b7cc87-fgscr\" (UID: \"9d33f110-3b40-4098-8075-2a42dba9c751\") " pod="openshift-ingress/payload-processing-e2e-inf-a-57b579-7994b7cc87-fgscr"
Sep 09 15:37:55.988140 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:55.987839    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh4kt\" (UniqueName: \"kubernetes.io/projected/9d33f110-3b40-4098-8075-2a42dba9c751-kube-api-access-nh4kt\") pod \"payload-processing-e2e-inf-a-57b579-7994b7cc87-fgscr\" (UID: \"9d33f110-3b40-4098-8075-2a42dba9c751\") " pod="openshift-ingress/payload-processing-e2e-inf-a-57b579-7994b7cc87-fgscr"
Sep 09 15:37:55.988140 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:55.987916    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm8gh\" (UniqueName: \"kubernetes.io/projected/015c0cd8-414e-4d52-aeee-c64c1588b315-kube-api-access-lm8gh\") pod \"payload-pre-processing-e2e-inf-a-57b579-58954c5b5d-dvsnc\" (UID: \"015c0cd8-414e-4d52-aeee-c64c1588b315\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-57b579-58954c5b5d-dvsnc"
Sep 09 15:37:56.088627 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:56.088591    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh4kt\" (UniqueName: \"kubernetes.io/projected/9d33f110-3b40-4098-8075-2a42dba9c751-kube-api-access-nh4kt\") pod \"payload-processing-e2e-inf-a-57b579-7994b7cc87-fgscr\" (UID: \"9d33f110-3b40-4098-8075-2a42dba9c751\") " pod="openshift-ingress/payload-processing-e2e-inf-a-57b579-7994b7cc87-fgscr"
Sep 09 15:37:56.088627 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:56.088631    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm8gh\" (UniqueName: \"kubernetes.io/projected/015c0cd8-414e-4d52-aeee-c64c1588b315-kube-api-access-lm8gh\") pod \"payload-pre-processing-e2e-inf-a-57b579-58954c5b5d-dvsnc\" (UID: \"015c0cd8-414e-4d52-aeee-c64c1588b315\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-57b579-58954c5b5d-dvsnc"
Sep 09 15:37:56.088893 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:56.088682    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/015c0cd8-414e-4d52-aeee-c64c1588b315-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-57b579-58954c5b5d-dvsnc\" (UID: \"015c0cd8-414e-4d52-aeee-c64c1588b315\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-57b579-58954c5b5d-dvsnc"
Sep 09 15:37:56.088893 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:56.088699    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d33f110-3b40-4098-8075-2a42dba9c751-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-57b579-7994b7cc87-fgscr\" (UID: \"9d33f110-3b40-4098-8075-2a42dba9c751\") " pod="openshift-ingress/payload-processing-e2e-inf-a-57b579-7994b7cc87-fgscr"
Sep 09 15:37:56.089610 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:56.089586    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d33f110-3b40-4098-8075-2a42dba9c751-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-57b579-7994b7cc87-fgscr\" (UID: \"9d33f110-3b40-4098-8075-2a42dba9c751\") " pod="openshift-ingress/payload-processing-e2e-inf-a-57b579-7994b7cc87-fgscr"
Sep 09 15:37:56.089664 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:56.089586    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/015c0cd8-414e-4d52-aeee-c64c1588b315-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-57b579-58954c5b5d-dvsnc\" (UID: \"015c0cd8-414e-4d52-aeee-c64c1588b315\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-57b579-58954c5b5d-dvsnc"
Sep 09 15:37:56.096790 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:56.096761    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh4kt\" (UniqueName: \"kubernetes.io/projected/9d33f110-3b40-4098-8075-2a42dba9c751-kube-api-access-nh4kt\") pod \"payload-processing-e2e-inf-a-57b579-7994b7cc87-fgscr\" (UID: \"9d33f110-3b40-4098-8075-2a42dba9c751\") " pod="openshift-ingress/payload-processing-e2e-inf-a-57b579-7994b7cc87-fgscr"
Sep 09 15:37:56.096901 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:56.096819    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm8gh\" (UniqueName: \"kubernetes.io/projected/015c0cd8-414e-4d52-aeee-c64c1588b315-kube-api-access-lm8gh\") pod \"payload-pre-processing-e2e-inf-a-57b579-58954c5b5d-dvsnc\" (UID: \"015c0cd8-414e-4d52-aeee-c64c1588b315\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-57b579-58954c5b5d-dvsnc"
Sep 09 15:37:56.194031 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:56.193994    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-57b579-58954c5b5d-dvsnc"
Sep 09 15:37:56.220542 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:56.220502    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-57b579-7994b7cc87-fgscr"
Sep 09 15:37:56.486814 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:56.486788    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-57b579-58954c5b5d-dvsnc"]
Sep 09 15:37:56.488807 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:37:56.488777    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod015c0cd8_414e_4d52_aeee_c64c1588b315.slice/crio-70bde9b2be9a03d8296cbaa8699ab021127fe7f9b68838d300623047905cfebc WatchSource:0}: Error finding container 70bde9b2be9a03d8296cbaa8699ab021127fe7f9b68838d300623047905cfebc: Status 404 returned error can't find the container with id 70bde9b2be9a03d8296cbaa8699ab021127fe7f9b68838d300623047905cfebc
Sep 09 15:37:56.533583 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:56.533560    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-57b579-7994b7cc87-fgscr"]
Sep 09 15:37:56.536693 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:37:56.536664    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d33f110_3b40_4098_8075_2a42dba9c751.slice/crio-506d56f436354d916a6bb5463b0919d1fb5051e47d0644a154a24aa1fa12f4ef WatchSource:0}: Error finding container 506d56f436354d916a6bb5463b0919d1fb5051e47d0644a154a24aa1fa12f4ef: Status 404 returned error can't find the container with id 506d56f436354d916a6bb5463b0919d1fb5051e47d0644a154a24aa1fa12f4ef
Sep 09 15:37:56.863019 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:56.862903    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-57b579-7994b7cc87-fgscr" event={"ID":"9d33f110-3b40-4098-8075-2a42dba9c751","Type":"ContainerStarted","Data":"4d74934ef89406a1c0b42f768da7324874405e0274ad76bf6855898c009000b6"}
Sep 09 15:37:56.863019 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:56.863003    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-57b579-7994b7cc87-fgscr" event={"ID":"9d33f110-3b40-4098-8075-2a42dba9c751","Type":"ContainerStarted","Data":"506d56f436354d916a6bb5463b0919d1fb5051e47d0644a154a24aa1fa12f4ef"}
Sep 09 15:37:56.863295 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:56.863033    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-57b579-7994b7cc87-fgscr"
Sep 09 15:37:56.865474 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:56.864912    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-57b579-58954c5b5d-dvsnc" event={"ID":"015c0cd8-414e-4d52-aeee-c64c1588b315","Type":"ContainerStarted","Data":"25de6eb9b21dde21f7a1f94cb9adaf868d5d5805edbbb052eee0e955e6503f00"}
Sep 09 15:37:56.865474 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:56.864965    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-57b579-58954c5b5d-dvsnc" event={"ID":"015c0cd8-414e-4d52-aeee-c64c1588b315","Type":"ContainerStarted","Data":"70bde9b2be9a03d8296cbaa8699ab021127fe7f9b68838d300623047905cfebc"}
Sep 09 15:37:56.865755 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:56.865696    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-57b579-58954c5b5d-dvsnc"
Sep 09 15:37:56.880286 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:56.880230    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-57b579-7994b7cc87-fgscr" podStartSLOduration=1.880213514 podStartE2EDuration="1.880213514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:37:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:37:56.879506274 +0000 UTC m=+1042.478400947" watchObservedRunningTime="2026-09-09 15:37:56.880213514 +0000 UTC m=+1042.479108158"
Sep 09 15:37:56.897137 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:37:56.897083    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-57b579-58954c5b5d-dvsnc" podStartSLOduration=1.897066464 podStartE2EDuration="1.897066464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:37:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:37:56.895632677 +0000 UTC m=+1042.494527323" watchObservedRunningTime="2026-09-09 15:37:56.897066464 +0000 UTC m=+1042.495961108"
Sep 09 15:38:02.459814 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:02.459759    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-8a43b2-69d656764f-f7m26"]
Sep 09 15:38:02.467190 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:02.467158    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-8a43b2-69d656764f-f7m26"
Sep 09 15:38:02.469515 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:02.469484    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-8a43b2-69d656764f-f7m26"]
Sep 09 15:38:02.477486 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:02.477457    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-8a43b2-dockercfg-9m5j2\""
Sep 09 15:38:02.478047 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:02.478025    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-8a43b2\""
Sep 09 15:38:02.580300 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:02.580254    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/125b5413-0dd4-459a-b430-e6ab4e57476f-plugins-config-volume\") pod \"payload-processing-e2e-autores-8a43b2-69d656764f-f7m26\" (UID: \"125b5413-0dd4-459a-b430-e6ab4e57476f\") " pod="openshift-ingress/payload-processing-e2e-autores-8a43b2-69d656764f-f7m26"
Sep 09 15:38:02.580942 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:02.580916    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtgl6\" (UniqueName: \"kubernetes.io/projected/125b5413-0dd4-459a-b430-e6ab4e57476f-kube-api-access-jtgl6\") pod \"payload-processing-e2e-autores-8a43b2-69d656764f-f7m26\" (UID: \"125b5413-0dd4-459a-b430-e6ab4e57476f\") " pod="openshift-ingress/payload-processing-e2e-autores-8a43b2-69d656764f-f7m26"
Sep 09 15:38:02.682327 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:02.682283    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/125b5413-0dd4-459a-b430-e6ab4e57476f-plugins-config-volume\") pod \"payload-processing-e2e-autores-8a43b2-69d656764f-f7m26\" (UID: \"125b5413-0dd4-459a-b430-e6ab4e57476f\") " pod="openshift-ingress/payload-processing-e2e-autores-8a43b2-69d656764f-f7m26"
Sep 09 15:38:02.682635 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:02.682613    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtgl6\" (UniqueName: \"kubernetes.io/projected/125b5413-0dd4-459a-b430-e6ab4e57476f-kube-api-access-jtgl6\") pod \"payload-processing-e2e-autores-8a43b2-69d656764f-f7m26\" (UID: \"125b5413-0dd4-459a-b430-e6ab4e57476f\") " pod="openshift-ingress/payload-processing-e2e-autores-8a43b2-69d656764f-f7m26"
Sep 09 15:38:02.684262 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:02.684123    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/125b5413-0dd4-459a-b430-e6ab4e57476f-plugins-config-volume\") pod \"payload-processing-e2e-autores-8a43b2-69d656764f-f7m26\" (UID: \"125b5413-0dd4-459a-b430-e6ab4e57476f\") " pod="openshift-ingress/payload-processing-e2e-autores-8a43b2-69d656764f-f7m26"
Sep 09 15:38:02.691561 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:02.691510    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtgl6\" (UniqueName: \"kubernetes.io/projected/125b5413-0dd4-459a-b430-e6ab4e57476f-kube-api-access-jtgl6\") pod \"payload-processing-e2e-autores-8a43b2-69d656764f-f7m26\" (UID: \"125b5413-0dd4-459a-b430-e6ab4e57476f\") " pod="openshift-ingress/payload-processing-e2e-autores-8a43b2-69d656764f-f7m26"
Sep 09 15:38:02.802401 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:02.802314    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-8a43b2-69d656764f-f7m26"
Sep 09 15:38:03.128974 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:03.128944    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-8a43b2-69d656764f-f7m26"]
Sep 09 15:38:03.130743 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:38:03.130706    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod125b5413_0dd4_459a_b430_e6ab4e57476f.slice/crio-137d1d282c3e94c8352dde1efb773d93c54829bda3c5a0d10f7cc121b8d870c4 WatchSource:0}: Error finding container 137d1d282c3e94c8352dde1efb773d93c54829bda3c5a0d10f7cc121b8d870c4: Status 404 returned error can't find the container with id 137d1d282c3e94c8352dde1efb773d93c54829bda3c5a0d10f7cc121b8d870c4
Sep 09 15:38:03.903714 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:03.903679    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-8a43b2-69d656764f-f7m26" event={"ID":"125b5413-0dd4-459a-b430-e6ab4e57476f","Type":"ContainerStarted","Data":"a181d0ce84a81286c8cf216eee7653b477e99f3bb4c492770b274c3ec924d71e"}
Sep 09 15:38:03.904110 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:03.903752    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-8a43b2-69d656764f-f7m26" event={"ID":"125b5413-0dd4-459a-b430-e6ab4e57476f","Type":"ContainerStarted","Data":"137d1d282c3e94c8352dde1efb773d93c54829bda3c5a0d10f7cc121b8d870c4"}
Sep 09 15:38:03.904264 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:03.904213    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-8a43b2-69d656764f-f7m26"
Sep 09 15:38:03.928711 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:03.928664    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-8a43b2-69d656764f-f7m26" podStartSLOduration=1.928647338 podStartE2EDuration="1.928647338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:38:03.927557901 +0000 UTC m=+1049.526452561" watchObservedRunningTime="2026-09-09 15:38:03.928647338 +0000 UTC m=+1049.527541982"
Sep 09 15:38:05.858856 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:05.858828    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-df30be-57ff546d6b-ts9mj"
Sep 09 15:38:06.868380 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:06.868341    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1e15c8-6f968d8478-8gkmw"
Sep 09 15:38:06.870949 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:06.870919    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-1e15c8-7c96479557-8f8jr"
Sep 09 15:38:07.872101 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:07.872072    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-57b579-7994b7cc87-fgscr"
Sep 09 15:38:08.876232 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:08.876198    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-57b579-58954c5b5d-dvsnc"
Sep 09 15:38:10.947836 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:10.947799    2553 generic.go:356] "Generic (PLEG): container finished" podID="5778704d-c2e2-4553-b8a5-0a7ebf84f9dc" containerID="b8833e8ec397320bddc2396cb9a39708c97e818883a51aea284024664971fa8b" exitCode=0
Sep 09 15:38:10.947836 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:10.947823    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv" event={"ID":"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc","Type":"ContainerDied","Data":"b8833e8ec397320bddc2396cb9a39708c97e818883a51aea284024664971fa8b"}
Sep 09 15:38:10.948336 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:10.948288    2553 scope.go:117] "RemoveContainer" containerID="b8833e8ec397320bddc2396cb9a39708c97e818883a51aea284024664971fa8b"
Sep 09 15:38:10.949546 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:10.949516    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:38:10.949642 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:10.949612    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:38:10.949703 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:10.949671    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:38:10.949782 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:10.949699    2553 generic.go:356] "Generic (PLEG): container finished" podID="3b0eec3f-27bc-4191-a148-4c5a8a531611" containerID="635dd9ae07ff8cfe25f6de4ddff68a105eab182d691bd46247c98ed82d369912" exitCode=0
Sep 09 15:38:10.949835 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:10.949775    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k" event={"ID":"3b0eec3f-27bc-4191-a148-4c5a8a531611","Type":"ContainerDied","Data":"635dd9ae07ff8cfe25f6de4ddff68a105eab182d691bd46247c98ed82d369912"}
Sep 09 15:38:10.950120 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:10.950106    2553 scope.go:117] "RemoveContainer" containerID="635dd9ae07ff8cfe25f6de4ddff68a105eab182d691bd46247c98ed82d369912"
Sep 09 15:38:10.951111 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:10.951087    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:38:10.951180 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:10.951159    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:38:10.951219 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:10.951190    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:38:11.960370 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:11.960334    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv" event={"ID":"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc","Type":"ContainerStarted","Data":"319dd4135318c95a948b2a0cf44a7df19c8a2f43dfe79a3b585be5c694798fc3"}
Sep 09 15:38:11.964576 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:11.964547    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k" event={"ID":"3b0eec3f-27bc-4191-a148-4c5a8a531611","Type":"ContainerStarted","Data":"5176c530d58c9649dba3d393258d041bfb3c6d023cdf557f42e60bd47dc2427d"}
Sep 09 15:38:11.971828 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:11.971756    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"
Sep 09 15:38:12.155366 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:12.155318    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"
Sep 09 15:38:12.155566 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:12.155403    2553 patch_prober.go:28] interesting pod/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body=
Sep 09 15:38:12.155566 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:12.155465    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv" podUID="5778704d-c2e2-4553-b8a5-0a7ebf84f9dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused"
Sep 09 15:38:12.979073 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:12.979031    2553 patch_prober.go:28] interesting pod/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Sep 09 15:38:12.979586 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:12.979094    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k" podUID="3b0eec3f-27bc-4191-a148-4c5a8a531611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Sep 09 15:38:13.155533 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:13.155490    2553 patch_prober.go:28] interesting pod/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body=
Sep 09 15:38:13.155843 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:13.155809    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv" podUID="5778704d-c2e2-4553-b8a5-0a7ebf84f9dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused"
Sep 09 15:38:13.972720 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:13.972676    2553 patch_prober.go:28] interesting pod/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Sep 09 15:38:13.973030 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:13.973000    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k" podUID="3b0eec3f-27bc-4191-a148-4c5a8a531611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Sep 09 15:38:14.154641 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:14.154603    2553 patch_prober.go:28] interesting pod/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body=
Sep 09 15:38:14.155229 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:14.155196    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv" podUID="5778704d-c2e2-4553-b8a5-0a7ebf84f9dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused"
Sep 09 15:38:14.972819 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:14.972785    2553 patch_prober.go:28] interesting pod/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Sep 09 15:38:14.973087 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:14.973059    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k" podUID="3b0eec3f-27bc-4191-a148-4c5a8a531611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Sep 09 15:38:15.154865 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:15.154818    2553 patch_prober.go:28] interesting pod/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body=
Sep 09 15:38:15.155366 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:15.154899    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv" podUID="5778704d-c2e2-4553-b8a5-0a7ebf84f9dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused"
Sep 09 15:38:15.917116 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:15.917083    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-8a43b2-69d656764f-f7m26"
Sep 09 15:38:15.978031 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:15.973721    2553 patch_prober.go:28] interesting pod/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Sep 09 15:38:15.978320 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:15.978288    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k" podUID="3b0eec3f-27bc-4191-a148-4c5a8a531611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Sep 09 15:38:16.155542 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:16.155501    2553 patch_prober.go:28] interesting pod/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body=
Sep 09 15:38:16.156174 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:16.156137    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv" podUID="5778704d-c2e2-4553-b8a5-0a7ebf84f9dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused"
Sep 09 15:38:16.971821 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:16.971784    2553 patch_prober.go:28] interesting pod/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Sep 09 15:38:16.972182 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:16.972068    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k" podUID="3b0eec3f-27bc-4191-a148-4c5a8a531611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Sep 09 15:38:17.154908 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:17.154868    2553 patch_prober.go:28] interesting pod/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body=
Sep 09 15:38:17.155081 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:17.154928    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv" podUID="5778704d-c2e2-4553-b8a5-0a7ebf84f9dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused"
Sep 09 15:38:17.976747 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:17.976702    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"
Sep 09 15:38:17.977227 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:17.977214    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"
Sep 09 15:38:17.977921 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:17.977899    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"
Sep 09 15:38:18.158526 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:18.158498    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"
Sep 09 15:38:18.158785 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:18.158764    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"
Sep 09 15:38:18.159500 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:18.159482    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"
Sep 09 15:38:29.552764 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:29.552645    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-267962-796f8bdb8b-pxs9m"]
Sep 09 15:38:29.557705 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:29.557681    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-267962-796f8bdb8b-pxs9m"
Sep 09 15:38:29.560495 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:29.560473    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-267962-dockercfg-cnx6l\""
Sep 09 15:38:29.560626 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:29.560506    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-267962\""
Sep 09 15:38:29.567624 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:29.567595    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-267962-796f8bdb8b-pxs9m"]
Sep 09 15:38:29.594024 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:29.593992    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c76a304-46ss8"]
Sep 09 15:38:29.597394 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:29.597371    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c76a304-46ss8"
Sep 09 15:38:29.600055 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:29.600028    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 15:38:29.600187 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:29.600098    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2zlsf\""
Sep 09 15:38:29.607208 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:29.606916    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c76a304-46ss8"]
Sep 09 15:38:29.673981 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:29.673944    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c55048a7-6b0b-4b63-99c8-cb695ed7217f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-267962-796f8bdb8b-pxs9m\" (UID: \"c55048a7-6b0b-4b63-99c8-cb695ed7217f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-267962-796f8bdb8b-pxs9m"
Sep 09 15:38:29.674161 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:29.674035    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljlc7\" (UniqueName: \"kubernetes.io/projected/c55048a7-6b0b-4b63-99c8-cb695ed7217f-kube-api-access-ljlc7\") pod \"payload-pre-processing-e2e-inf-b-267962-796f8bdb8b-pxs9m\" (UID: \"c55048a7-6b0b-4b63-99c8-cb695ed7217f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-267962-796f8bdb8b-pxs9m"
Sep 09 15:38:29.775100 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:29.775054    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljlc7\" (UniqueName: \"kubernetes.io/projected/c55048a7-6b0b-4b63-99c8-cb695ed7217f-kube-api-access-ljlc7\") pod \"payload-pre-processing-e2e-inf-b-267962-796f8bdb8b-pxs9m\" (UID: \"c55048a7-6b0b-4b63-99c8-cb695ed7217f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-267962-796f8bdb8b-pxs9m"
Sep 09 15:38:29.775284 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:29.775142    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ac0344c-301b-4eb6-825e-3c9c0e0adf2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1c76a304-46ss8\" (UID: \"1ac0344c-301b-4eb6-825e-3c9c0e0adf2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c76a304-46ss8"
Sep 09 15:38:29.775284 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:29.775196    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c55048a7-6b0b-4b63-99c8-cb695ed7217f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-267962-796f8bdb8b-pxs9m\" (UID: \"c55048a7-6b0b-4b63-99c8-cb695ed7217f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-267962-796f8bdb8b-pxs9m"
Sep 09 15:38:29.775994 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:29.775976    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c55048a7-6b0b-4b63-99c8-cb695ed7217f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-267962-796f8bdb8b-pxs9m\" (UID: \"c55048a7-6b0b-4b63-99c8-cb695ed7217f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-267962-796f8bdb8b-pxs9m"
Sep 09 15:38:29.783140 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:29.783119    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljlc7\" (UniqueName: \"kubernetes.io/projected/c55048a7-6b0b-4b63-99c8-cb695ed7217f-kube-api-access-ljlc7\") pod \"payload-pre-processing-e2e-inf-b-267962-796f8bdb8b-pxs9m\" (UID: \"c55048a7-6b0b-4b63-99c8-cb695ed7217f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-267962-796f8bdb8b-pxs9m"
Sep 09 15:38:29.870573 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:29.870473    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-267962-796f8bdb8b-pxs9m"
Sep 09 15:38:29.875983 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:29.875957    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ac0344c-301b-4eb6-825e-3c9c0e0adf2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1c76a304-46ss8\" (UID: \"1ac0344c-301b-4eb6-825e-3c9c0e0adf2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c76a304-46ss8"
Sep 09 15:38:29.876517 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:29.876497    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ac0344c-301b-4eb6-825e-3c9c0e0adf2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1c76a304-46ss8\" (UID: \"1ac0344c-301b-4eb6-825e-3c9c0e0adf2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c76a304-46ss8"
Sep 09 15:38:29.912167 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:29.912130    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c76a304-46ss8"
Sep 09 15:38:30.150566 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:30.150541    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-267962-796f8bdb8b-pxs9m"]
Sep 09 15:38:30.152775 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:38:30.152717    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc55048a7_6b0b_4b63_99c8_cb695ed7217f.slice/crio-e721d29be672442355318c4dd239bd9d378a24025481e134cf236f432694c183 WatchSource:0}: Error finding container e721d29be672442355318c4dd239bd9d378a24025481e134cf236f432694c183: Status 404 returned error can't find the container with id e721d29be672442355318c4dd239bd9d378a24025481e134cf236f432694c183
Sep 09 15:38:30.202574 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:30.202551    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c76a304-46ss8"]
Sep 09 15:38:30.204417 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:38:30.204394    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ac0344c_301b_4eb6_825e_3c9c0e0adf2e.slice/crio-1de4c07ac777e25671aa367216ce7ac821bde9b4c0816bce073791546dfafa3f WatchSource:0}: Error finding container 1de4c07ac777e25671aa367216ce7ac821bde9b4c0816bce073791546dfafa3f: Status 404 returned error can't find the container with id 1de4c07ac777e25671aa367216ce7ac821bde9b4c0816bce073791546dfafa3f
Sep 09 15:38:31.080015 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:31.079974    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-267962-796f8bdb8b-pxs9m" event={"ID":"c55048a7-6b0b-4b63-99c8-cb695ed7217f","Type":"ContainerStarted","Data":"97e09bab7be1a41efe33b3625f358b1087640dec7634b3cd8b473f45a114a292"}
Sep 09 15:38:31.080507 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:31.080043    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-267962-796f8bdb8b-pxs9m" event={"ID":"c55048a7-6b0b-4b63-99c8-cb695ed7217f","Type":"ContainerStarted","Data":"e721d29be672442355318c4dd239bd9d378a24025481e134cf236f432694c183"}
Sep 09 15:38:31.080507 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:31.080195    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-267962-796f8bdb8b-pxs9m"
Sep 09 15:38:31.081393 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:31.081363    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c76a304-46ss8" event={"ID":"1ac0344c-301b-4eb6-825e-3c9c0e0adf2e","Type":"ContainerStarted","Data":"1de4c07ac777e25671aa367216ce7ac821bde9b4c0816bce073791546dfafa3f"}
Sep 09 15:38:31.096147 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:31.096092    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-267962-796f8bdb8b-pxs9m" podStartSLOduration=2.096076312 podStartE2EDuration="2.096076312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:38:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:38:31.094839638 +0000 UTC m=+1076.693734282" watchObservedRunningTime="2026-09-09 15:38:31.096076312 +0000 UTC m=+1076.694970956"
Sep 09 15:38:33.094096 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:33.094051    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c76a304-46ss8" event={"ID":"1ac0344c-301b-4eb6-825e-3c9c0e0adf2e","Type":"ContainerStarted","Data":"12b981f6562781df154b7148d41d393180d2538c08f5af66921367f4bcc31a43"}
Sep 09 15:38:33.110643 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:33.110543    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c76a304-46ss8" podStartSLOduration=1.75621678 podStartE2EDuration="4.11052723s" totalImagesPullingTime="2.35431045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:38:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:38:33.108003612 +0000 UTC m=+1078.706898248" watchObservedRunningTime="2026-09-09 15:38:33.11052723 +0000 UTC m=+1078.709421870"
Sep 09 15:38:34.100244 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:34.100209    2553 generic.go:356] "Generic (PLEG): container finished" podID="1ac0344c-301b-4eb6-825e-3c9c0e0adf2e" containerID="12b981f6562781df154b7148d41d393180d2538c08f5af66921367f4bcc31a43" exitCode=0
Sep 09 15:38:34.100631 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:34.100295    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c76a304-46ss8" event={"ID":"1ac0344c-301b-4eb6-825e-3c9c0e0adf2e","Type":"ContainerDied","Data":"12b981f6562781df154b7148d41d393180d2538c08f5af66921367f4bcc31a43"}
Sep 09 15:38:35.239548 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:35.239518    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c76a304-46ss8"
Sep 09 15:38:35.429221 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:35.429185    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ac0344c-301b-4eb6-825e-3c9c0e0adf2e-maas-api-service-ca\") pod \"1ac0344c-301b-4eb6-825e-3c9c0e0adf2e\" (UID: \"1ac0344c-301b-4eb6-825e-3c9c0e0adf2e\") "
Sep 09 15:38:35.429563 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:35.429540    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ac0344c-301b-4eb6-825e-3c9c0e0adf2e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1ac0344c-301b-4eb6-825e-3c9c0e0adf2e" (UID: "1ac0344c-301b-4eb6-825e-3c9c0e0adf2e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:38:35.530065 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:35.530030    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ac0344c-301b-4eb6-825e-3c9c0e0adf2e-maas-api-service-ca\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:38:36.117402 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:36.117364    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c76a304-46ss8" event={"ID":"1ac0344c-301b-4eb6-825e-3c9c0e0adf2e","Type":"ContainerDied","Data":"1de4c07ac777e25671aa367216ce7ac821bde9b4c0816bce073791546dfafa3f"}
Sep 09 15:38:36.117402 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:36.117405    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c76a304-46ss8"
Sep 09 15:38:36.117617 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:36.117440    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1de4c07ac777e25671aa367216ce7ac821bde9b4c0816bce073791546dfafa3f"
Sep 09 15:38:42.089773 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:42.089745    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-267962-796f8bdb8b-pxs9m"
Sep 09 15:38:46.547597 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:46.547555    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c76a304-46ss8"]
Sep 09 15:38:46.552614 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:46.552580    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c76a304-46ss8"]
Sep 09 15:38:46.962395 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:38:46.962359    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ac0344c-301b-4eb6-825e-3c9c0e0adf2e" path="/var/lib/kubelet/pods/1ac0344c-301b-4eb6-825e-3c9c0e0adf2e/volumes"
Sep 09 15:39:11.164494 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:11.164453    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2"]
Sep 09 15:39:11.164983 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:11.164870    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ac0344c-301b-4eb6-825e-3c9c0e0adf2e" containerName="revoke-keys"
Sep 09 15:39:11.164983 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:11.164882    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ac0344c-301b-4eb6-825e-3c9c0e0adf2e" containerName="revoke-keys"
Sep 09 15:39:11.164983 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:11.164952    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ac0344c-301b-4eb6-825e-3c9c0e0adf2e" containerName="revoke-keys"
Sep 09 15:39:11.172694 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:11.172666    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1e15c8/simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2"
Sep 09 15:39:11.175310 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:11.175280    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2"]
Sep 09 15:39:11.175475 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:11.175454    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-1e15c8\"/\"kube-root-ca.crt\""
Sep 09 15:39:11.175527 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:11.175504    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1e15c8\"/\"simulator-w4-1e15c8-kserve-self-signed-certs\""
Sep 09 15:39:11.175580 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:11.175509    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-1e15c8\"/\"openshift-service-ca.crt\""
Sep 09 15:39:11.176497 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:11.176476    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1e15c8\"/\"default-dockercfg-qxwlq\""
Sep 09 15:39:11.252257 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:11.252219    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc074101-6a71-45f8-8ba8-24a224e135bd-home\") pod \"simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2\" (UID: \"cc074101-6a71-45f8-8ba8-24a224e135bd\") " pod="e2e-models-e2e-worker-w4-1e15c8/simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2"
Sep 09 15:39:11.252437 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:11.252268    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc074101-6a71-45f8-8ba8-24a224e135bd-model-cache\") pod \"simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2\" (UID: \"cc074101-6a71-45f8-8ba8-24a224e135bd\") " pod="e2e-models-e2e-worker-w4-1e15c8/simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2"
Sep 09 15:39:11.252437 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:11.252332    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc074101-6a71-45f8-8ba8-24a224e135bd-dshm\") pod \"simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2\" (UID: \"cc074101-6a71-45f8-8ba8-24a224e135bd\") " pod="e2e-models-e2e-worker-w4-1e15c8/simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2"
Sep 09 15:39:11.252437 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:11.252368    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc074101-6a71-45f8-8ba8-24a224e135bd-tls-certs\") pod \"simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2\" (UID: \"cc074101-6a71-45f8-8ba8-24a224e135bd\") " pod="e2e-models-e2e-worker-w4-1e15c8/simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2"
Sep 09 15:39:11.252437 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:11.252394    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmvkh\" (UniqueName: \"kubernetes.io/projected/cc074101-6a71-45f8-8ba8-24a224e135bd-kube-api-access-pmvkh\") pod \"simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2\" (UID: \"cc074101-6a71-45f8-8ba8-24a224e135bd\") " pod="e2e-models-e2e-worker-w4-1e15c8/simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2"
Sep 09 15:39:11.252594 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:11.252478    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc074101-6a71-45f8-8ba8-24a224e135bd-tmp-dir\") pod \"simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2\" (UID: \"cc074101-6a71-45f8-8ba8-24a224e135bd\") " pod="e2e-models-e2e-worker-w4-1e15c8/simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2"
Sep 09 15:39:11.353194 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:11.353154    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc074101-6a71-45f8-8ba8-24a224e135bd-tmp-dir\") pod \"simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2\" (UID: \"cc074101-6a71-45f8-8ba8-24a224e135bd\") " pod="e2e-models-e2e-worker-w4-1e15c8/simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2"
Sep 09 15:39:11.353363 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:11.353206    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc074101-6a71-45f8-8ba8-24a224e135bd-home\") pod \"simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2\" (UID: \"cc074101-6a71-45f8-8ba8-24a224e135bd\") " pod="e2e-models-e2e-worker-w4-1e15c8/simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2"
Sep 09 15:39:11.353363 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:11.353345    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc074101-6a71-45f8-8ba8-24a224e135bd-model-cache\") pod \"simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2\" (UID: \"cc074101-6a71-45f8-8ba8-24a224e135bd\") " pod="e2e-models-e2e-worker-w4-1e15c8/simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2"
Sep 09 15:39:11.353456 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:11.353430    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc074101-6a71-45f8-8ba8-24a224e135bd-dshm\") pod \"simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2\" (UID: \"cc074101-6a71-45f8-8ba8-24a224e135bd\") " pod="e2e-models-e2e-worker-w4-1e15c8/simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2"
Sep 09 15:39:11.353506 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:11.353473    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc074101-6a71-45f8-8ba8-24a224e135bd-tls-certs\") pod \"simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2\" (UID: \"cc074101-6a71-45f8-8ba8-24a224e135bd\") " pod="e2e-models-e2e-worker-w4-1e15c8/simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2"
Sep 09 15:39:11.353559 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:11.353503    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmvkh\" (UniqueName: \"kubernetes.io/projected/cc074101-6a71-45f8-8ba8-24a224e135bd-kube-api-access-pmvkh\") pod \"simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2\" (UID: \"cc074101-6a71-45f8-8ba8-24a224e135bd\") " pod="e2e-models-e2e-worker-w4-1e15c8/simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2"
Sep 09 15:39:11.353608 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:11.353587    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc074101-6a71-45f8-8ba8-24a224e135bd-home\") pod \"simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2\" (UID: \"cc074101-6a71-45f8-8ba8-24a224e135bd\") " pod="e2e-models-e2e-worker-w4-1e15c8/simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2"
Sep 09 15:39:11.353660 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:11.353603    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc074101-6a71-45f8-8ba8-24a224e135bd-tmp-dir\") pod \"simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2\" (UID: \"cc074101-6a71-45f8-8ba8-24a224e135bd\") " pod="e2e-models-e2e-worker-w4-1e15c8/simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2"
Sep 09 15:39:11.353838 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:11.353817    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc074101-6a71-45f8-8ba8-24a224e135bd-model-cache\") pod \"simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2\" (UID: \"cc074101-6a71-45f8-8ba8-24a224e135bd\") " pod="e2e-models-e2e-worker-w4-1e15c8/simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2"
Sep 09 15:39:11.355837 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:11.355806    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc074101-6a71-45f8-8ba8-24a224e135bd-dshm\") pod \"simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2\" (UID: \"cc074101-6a71-45f8-8ba8-24a224e135bd\") " pod="e2e-models-e2e-worker-w4-1e15c8/simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2"
Sep 09 15:39:11.356048 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:11.356027    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc074101-6a71-45f8-8ba8-24a224e135bd-tls-certs\") pod \"simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2\" (UID: \"cc074101-6a71-45f8-8ba8-24a224e135bd\") " pod="e2e-models-e2e-worker-w4-1e15c8/simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2"
Sep 09 15:39:11.364847 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:11.364821    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmvkh\" (UniqueName: \"kubernetes.io/projected/cc074101-6a71-45f8-8ba8-24a224e135bd-kube-api-access-pmvkh\") pod \"simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2\" (UID: \"cc074101-6a71-45f8-8ba8-24a224e135bd\") " pod="e2e-models-e2e-worker-w4-1e15c8/simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2"
Sep 09 15:39:11.483218 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:11.483177    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1e15c8/simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2"
Sep 09 15:39:11.762898 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:11.762862    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2"]
Sep 09 15:39:11.764298 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:39:11.764261    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc074101_6a71_45f8_8ba8_24a224e135bd.slice/crio-ecc08c463ffb4f6f617c36186e19ac07462370fa173f95caef341dc91939dd03 WatchSource:0}: Error finding container ecc08c463ffb4f6f617c36186e19ac07462370fa173f95caef341dc91939dd03: Status 404 returned error can't find the container with id ecc08c463ffb4f6f617c36186e19ac07462370fa173f95caef341dc91939dd03
Sep 09 15:39:12.298216 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:12.298183    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1e15c8/simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2" event={"ID":"cc074101-6a71-45f8-8ba8-24a224e135bd","Type":"ContainerStarted","Data":"fcd54b86b765bea721385a0cbb5c955c725661d2077f15b8ff4240b09c9a649f"}
Sep 09 15:39:12.298602 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:12.298225    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1e15c8/simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2" event={"ID":"cc074101-6a71-45f8-8ba8-24a224e135bd","Type":"ContainerStarted","Data":"ecc08c463ffb4f6f617c36186e19ac07462370fa173f95caef341dc91939dd03"}
Sep 09 15:39:12.316245 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:12.316193    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-1e15c8/simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2" podStartSLOduration=1.316174673 podStartE2EDuration="1.316174673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:39:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:39:12.313893816 +0000 UTC m=+1117.912788461" watchObservedRunningTime="2026-09-09 15:39:12.316174673 +0000 UTC m=+1117.915069317"
Sep 09 15:39:15.213262 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:15.213229    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a7ffc761-ps57b"]
Sep 09 15:39:15.216894 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:15.216877    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a7ffc761-ps57b"
Sep 09 15:39:15.219511 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:15.219487    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2zlsf\""
Sep 09 15:39:15.219636 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:15.219529    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 15:39:15.223447 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:15.223419    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a7ffc761-ps57b"]
Sep 09 15:39:15.388295 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:15.388256    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c3bced2-4030-4efb-916c-b55b17fe77df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a7ffc761-ps57b\" (UID: \"7c3bced2-4030-4efb-916c-b55b17fe77df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a7ffc761-ps57b"
Sep 09 15:39:15.489527 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:15.489427    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c3bced2-4030-4efb-916c-b55b17fe77df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a7ffc761-ps57b\" (UID: \"7c3bced2-4030-4efb-916c-b55b17fe77df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a7ffc761-ps57b"
Sep 09 15:39:15.490109 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:15.490089    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c3bced2-4030-4efb-916c-b55b17fe77df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a7ffc761-ps57b\" (UID: \"7c3bced2-4030-4efb-916c-b55b17fe77df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a7ffc761-ps57b"
Sep 09 15:39:15.530110 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:15.530078    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a7ffc761-ps57b"
Sep 09 15:39:15.800086 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:15.800057    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a7ffc761-ps57b"]
Sep 09 15:39:15.802441 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:39:15.802406    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c3bced2_4030_4efb_916c_b55b17fe77df.slice/crio-9457bfbb1299f31326aad1bb7396a76591c8c83707f4ac7e20b354bca3fb173a WatchSource:0}: Error finding container 9457bfbb1299f31326aad1bb7396a76591c8c83707f4ac7e20b354bca3fb173a: Status 404 returned error can't find the container with id 9457bfbb1299f31326aad1bb7396a76591c8c83707f4ac7e20b354bca3fb173a
Sep 09 15:39:16.318843 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:16.318805    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a7ffc761-ps57b" event={"ID":"7c3bced2-4030-4efb-916c-b55b17fe77df","Type":"ContainerStarted","Data":"8736831e8494a4fb58474fbbef594ce62f7395faff9a1e7b8001705d80ca7df1"}
Sep 09 15:39:16.319232 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:16.318860    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a7ffc761-ps57b" event={"ID":"7c3bced2-4030-4efb-916c-b55b17fe77df","Type":"ContainerStarted","Data":"9457bfbb1299f31326aad1bb7396a76591c8c83707f4ac7e20b354bca3fb173a"}
Sep 09 15:39:16.334920 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:16.334863    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a7ffc761-ps57b" podStartSLOduration=1.334845797 podStartE2EDuration="1.334845797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:39:16.333619044 +0000 UTC m=+1121.932513688" watchObservedRunningTime="2026-09-09 15:39:16.334845797 +0000 UTC m=+1121.933740441"
Sep 09 15:39:17.325424 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:17.325388    2553 generic.go:356] "Generic (PLEG): container finished" podID="7c3bced2-4030-4efb-916c-b55b17fe77df" containerID="8736831e8494a4fb58474fbbef594ce62f7395faff9a1e7b8001705d80ca7df1" exitCode=0
Sep 09 15:39:17.325835 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:17.325478    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a7ffc761-ps57b" event={"ID":"7c3bced2-4030-4efb-916c-b55b17fe77df","Type":"ContainerDied","Data":"8736831e8494a4fb58474fbbef594ce62f7395faff9a1e7b8001705d80ca7df1"}
Sep 09 15:39:18.483019 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:18.482987    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a7ffc761-ps57b"
Sep 09 15:39:18.617317 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:18.617226    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c3bced2-4030-4efb-916c-b55b17fe77df-maas-api-service-ca\") pod \"7c3bced2-4030-4efb-916c-b55b17fe77df\" (UID: \"7c3bced2-4030-4efb-916c-b55b17fe77df\") "
Sep 09 15:39:18.617578 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:18.617556    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c3bced2-4030-4efb-916c-b55b17fe77df-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7c3bced2-4030-4efb-916c-b55b17fe77df" (UID: "7c3bced2-4030-4efb-916c-b55b17fe77df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:39:18.718791 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:18.718749    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c3bced2-4030-4efb-916c-b55b17fe77df-maas-api-service-ca\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:39:19.337690 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:19.337604    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a7ffc761-ps57b" event={"ID":"7c3bced2-4030-4efb-916c-b55b17fe77df","Type":"ContainerDied","Data":"9457bfbb1299f31326aad1bb7396a76591c8c83707f4ac7e20b354bca3fb173a"}
Sep 09 15:39:19.337690 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:19.337654    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9457bfbb1299f31326aad1bb7396a76591c8c83707f4ac7e20b354bca3fb173a"
Sep 09 15:39:19.337690 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:19.337655    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a7ffc761-ps57b"
Sep 09 15:39:21.171854 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:21.171819    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a7ffc761-ps57b"]
Sep 09 15:39:21.173665 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:21.173640    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a7ffc761-ps57b"]
Sep 09 15:39:21.484176 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:21.484075    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1e15c8/simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2"
Sep 09 15:39:21.484427 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:21.484398    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-1e15c8/simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2"
Sep 09 15:39:21.497206 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:21.497175    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-1e15c8/simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2"
Sep 09 15:39:22.364596 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:22.364567    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1e15c8/simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2"
Sep 09 15:39:22.962062 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:22.962029    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c3bced2-4030-4efb-916c-b55b17fe77df" path="/var/lib/kubelet/pods/7c3bced2-4030-4efb-916c-b55b17fe77df/volumes"
Sep 09 15:39:50.062018 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:50.061985    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f1fbd128-zlfht"]
Sep 09 15:39:50.062490 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:50.062333    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c3bced2-4030-4efb-916c-b55b17fe77df" containerName="revoke-keys"
Sep 09 15:39:50.062490 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:50.062344    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c3bced2-4030-4efb-916c-b55b17fe77df" containerName="revoke-keys"
Sep 09 15:39:50.062490 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:50.062421    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c3bced2-4030-4efb-916c-b55b17fe77df" containerName="revoke-keys"
Sep 09 15:39:50.065479 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:50.065464    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f1fbd128-zlfht"
Sep 09 15:39:50.067835 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:50.067815    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2zlsf\""
Sep 09 15:39:50.068054 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:50.068040    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 15:39:50.073700 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:50.073676    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f1fbd128-zlfht"]
Sep 09 15:39:50.211380 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:50.211339    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f204414-9ddf-4fc3-94a4-d292a14e1c37-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f1fbd128-zlfht\" (UID: \"1f204414-9ddf-4fc3-94a4-d292a14e1c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f1fbd128-zlfht"
Sep 09 15:39:50.312560 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:50.312455    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f204414-9ddf-4fc3-94a4-d292a14e1c37-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f1fbd128-zlfht\" (UID: \"1f204414-9ddf-4fc3-94a4-d292a14e1c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f1fbd128-zlfht"
Sep 09 15:39:50.313131 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:50.313108    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f204414-9ddf-4fc3-94a4-d292a14e1c37-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f1fbd128-zlfht\" (UID: \"1f204414-9ddf-4fc3-94a4-d292a14e1c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f1fbd128-zlfht"
Sep 09 15:39:50.377597 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:50.377558    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f1fbd128-zlfht"
Sep 09 15:39:50.647094 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:50.647065    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f1fbd128-zlfht"]
Sep 09 15:39:50.648518 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:39:50.648487    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f204414_9ddf_4fc3_94a4_d292a14e1c37.slice/crio-1cf07feaea82267fa17d58bb07f73c0a9539b09d810a03134bc90af20b9a282c WatchSource:0}: Error finding container 1cf07feaea82267fa17d58bb07f73c0a9539b09d810a03134bc90af20b9a282c: Status 404 returned error can't find the container with id 1cf07feaea82267fa17d58bb07f73c0a9539b09d810a03134bc90af20b9a282c
Sep 09 15:39:51.501674 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:51.501581    2553 generic.go:356] "Generic (PLEG): container finished" podID="1f204414-9ddf-4fc3-94a4-d292a14e1c37" containerID="0495d7a6fb15b99dc903a318ed01a3dbd22a692808e0eefff16abe5acd8f653c" exitCode=0
Sep 09 15:39:51.501674 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:51.501662    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f1fbd128-zlfht" event={"ID":"1f204414-9ddf-4fc3-94a4-d292a14e1c37","Type":"ContainerDied","Data":"0495d7a6fb15b99dc903a318ed01a3dbd22a692808e0eefff16abe5acd8f653c"}
Sep 09 15:39:51.502071 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:51.501714    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f1fbd128-zlfht" event={"ID":"1f204414-9ddf-4fc3-94a4-d292a14e1c37","Type":"ContainerStarted","Data":"1cf07feaea82267fa17d58bb07f73c0a9539b09d810a03134bc90af20b9a282c"}
Sep 09 15:39:52.640936 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:52.640911    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f1fbd128-zlfht"
Sep 09 15:39:52.736579 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:52.736537    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f204414-9ddf-4fc3-94a4-d292a14e1c37-maas-api-service-ca\") pod \"1f204414-9ddf-4fc3-94a4-d292a14e1c37\" (UID: \"1f204414-9ddf-4fc3-94a4-d292a14e1c37\") "
Sep 09 15:39:52.736945 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:52.736920    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f204414-9ddf-4fc3-94a4-d292a14e1c37-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1f204414-9ddf-4fc3-94a4-d292a14e1c37" (UID: "1f204414-9ddf-4fc3-94a4-d292a14e1c37"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:39:52.837475 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:52.837377    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f204414-9ddf-4fc3-94a4-d292a14e1c37-maas-api-service-ca\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:39:53.513122 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:53.513090    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f1fbd128-zlfht"
Sep 09 15:39:53.513306 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:53.513121    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f1fbd128-zlfht" event={"ID":"1f204414-9ddf-4fc3-94a4-d292a14e1c37","Type":"ContainerDied","Data":"1cf07feaea82267fa17d58bb07f73c0a9539b09d810a03134bc90af20b9a282c"}
Sep 09 15:39:53.513306 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:53.513164    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cf07feaea82267fa17d58bb07f73c0a9539b09d810a03134bc90af20b9a282c"
Sep 09 15:39:56.161211 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:56.161176    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f1fbd128-zlfht"]
Sep 09 15:39:56.164029 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:56.164004    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f1fbd128-zlfht"]
Sep 09 15:39:56.962176 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:39:56.962100    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f204414-9ddf-4fc3-94a4-d292a14e1c37" path="/var/lib/kubelet/pods/1f204414-9ddf-4fc3-94a4-d292a14e1c37/volumes"
Sep 09 15:40:02.298778 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:02.295946    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c348b0d2-769b566b6f-qr9sm"]
Sep 09 15:40:02.298778 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:02.296895    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f204414-9ddf-4fc3-94a4-d292a14e1c37" containerName="revoke-keys"
Sep 09 15:40:02.298778 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:02.296917    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f204414-9ddf-4fc3-94a4-d292a14e1c37" containerName="revoke-keys"
Sep 09 15:40:02.298778 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:02.297117    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f204414-9ddf-4fc3-94a4-d292a14e1c37" containerName="revoke-keys"
Sep 09 15:40:02.301383 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:02.301356    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c348b0d2-769b566b6f-qr9sm"
Sep 09 15:40:02.304921 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:02.304897    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-c348b0d2\""
Sep 09 15:40:02.305049 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:02.305033    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-c348b0d2-dockercfg-l7hbd\""
Sep 09 15:40:02.314476 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:02.314450    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c348b0d2-769b566b6f-qr9sm"]
Sep 09 15:40:02.432031 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:02.431992    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/679e994b-fb9a-41ca-9d21-38d38d656db9-plugins-config-volume\") pod \"payload-processing-e2e-derive-c348b0d2-769b566b6f-qr9sm\" (UID: \"679e994b-fb9a-41ca-9d21-38d38d656db9\") " pod="openshift-ingress/payload-processing-e2e-derive-c348b0d2-769b566b6f-qr9sm"
Sep 09 15:40:02.432208 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:02.432057    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4svms\" (UniqueName: \"kubernetes.io/projected/679e994b-fb9a-41ca-9d21-38d38d656db9-kube-api-access-4svms\") pod \"payload-processing-e2e-derive-c348b0d2-769b566b6f-qr9sm\" (UID: \"679e994b-fb9a-41ca-9d21-38d38d656db9\") " pod="openshift-ingress/payload-processing-e2e-derive-c348b0d2-769b566b6f-qr9sm"
Sep 09 15:40:02.533283 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:02.533242    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/679e994b-fb9a-41ca-9d21-38d38d656db9-plugins-config-volume\") pod \"payload-processing-e2e-derive-c348b0d2-769b566b6f-qr9sm\" (UID: \"679e994b-fb9a-41ca-9d21-38d38d656db9\") " pod="openshift-ingress/payload-processing-e2e-derive-c348b0d2-769b566b6f-qr9sm"
Sep 09 15:40:02.533457 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:02.533304    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4svms\" (UniqueName: \"kubernetes.io/projected/679e994b-fb9a-41ca-9d21-38d38d656db9-kube-api-access-4svms\") pod \"payload-processing-e2e-derive-c348b0d2-769b566b6f-qr9sm\" (UID: \"679e994b-fb9a-41ca-9d21-38d38d656db9\") " pod="openshift-ingress/payload-processing-e2e-derive-c348b0d2-769b566b6f-qr9sm"
Sep 09 15:40:02.534029 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:02.534007    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/679e994b-fb9a-41ca-9d21-38d38d656db9-plugins-config-volume\") pod \"payload-processing-e2e-derive-c348b0d2-769b566b6f-qr9sm\" (UID: \"679e994b-fb9a-41ca-9d21-38d38d656db9\") " pod="openshift-ingress/payload-processing-e2e-derive-c348b0d2-769b566b6f-qr9sm"
Sep 09 15:40:02.547633 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:02.547598    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4svms\" (UniqueName: \"kubernetes.io/projected/679e994b-fb9a-41ca-9d21-38d38d656db9-kube-api-access-4svms\") pod \"payload-processing-e2e-derive-c348b0d2-769b566b6f-qr9sm\" (UID: \"679e994b-fb9a-41ca-9d21-38d38d656db9\") " pod="openshift-ingress/payload-processing-e2e-derive-c348b0d2-769b566b6f-qr9sm"
Sep 09 15:40:02.611722 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:02.611631    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c348b0d2-769b566b6f-qr9sm"
Sep 09 15:40:02.886001 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:02.885974    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c348b0d2-769b566b6f-qr9sm"]
Sep 09 15:40:02.887841 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:40:02.887812    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod679e994b_fb9a_41ca_9d21_38d38d656db9.slice/crio-71bc934aee7244eb399929a163068e3c91ed2a5e08517a1d9e4e8a20dcbaa482 WatchSource:0}: Error finding container 71bc934aee7244eb399929a163068e3c91ed2a5e08517a1d9e4e8a20dcbaa482: Status 404 returned error can't find the container with id 71bc934aee7244eb399929a163068e3c91ed2a5e08517a1d9e4e8a20dcbaa482
Sep 09 15:40:03.573062 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:03.573021    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c348b0d2-769b566b6f-qr9sm" event={"ID":"679e994b-fb9a-41ca-9d21-38d38d656db9","Type":"ContainerStarted","Data":"9935d35841d72f104e299fc3e4c44514d690765043a0a38d9a6ec3de27b7097d"}
Sep 09 15:40:03.573459 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:03.573088    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c348b0d2-769b566b6f-qr9sm" event={"ID":"679e994b-fb9a-41ca-9d21-38d38d656db9","Type":"ContainerStarted","Data":"71bc934aee7244eb399929a163068e3c91ed2a5e08517a1d9e4e8a20dcbaa482"}
Sep 09 15:40:03.573459 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:03.573230    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-c348b0d2-769b566b6f-qr9sm"
Sep 09 15:40:03.590323 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:03.590270    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-c348b0d2-769b566b6f-qr9sm" podStartSLOduration=1.590253767 podStartE2EDuration="1.590253767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:40:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:40:03.58906141 +0000 UTC m=+1169.187956054" watchObservedRunningTime="2026-09-09 15:40:03.590253767 +0000 UTC m=+1169.189148410"
Sep 09 15:40:08.869624 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:08.869587    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8a43b2-kxk6d"]
Sep 09 15:40:08.873071 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:08.873053    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8a43b2-kxk6d"
Sep 09 15:40:08.875546 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:08.875522    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 15:40:08.875546 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:08.875525    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2zlsf\""
Sep 09 15:40:08.879464 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:08.879439    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8a43b2-kxk6d"]
Sep 09 15:40:08.996698 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:08.996659    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdd81934-3be1-4608-a549-2bf0edf7674a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-8a43b2-kxk6d\" (UID: \"fdd81934-3be1-4608-a549-2bf0edf7674a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8a43b2-kxk6d"
Sep 09 15:40:09.098041 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:09.098000    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdd81934-3be1-4608-a549-2bf0edf7674a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-8a43b2-kxk6d\" (UID: \"fdd81934-3be1-4608-a549-2bf0edf7674a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8a43b2-kxk6d"
Sep 09 15:40:09.098643 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:09.098623    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdd81934-3be1-4608-a549-2bf0edf7674a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-8a43b2-kxk6d\" (UID: \"fdd81934-3be1-4608-a549-2bf0edf7674a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8a43b2-kxk6d"
Sep 09 15:40:09.185830 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:09.185800    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8a43b2-kxk6d"
Sep 09 15:40:09.455592 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:09.455565    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8a43b2-kxk6d"]
Sep 09 15:40:09.457757 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:40:09.457716    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdd81934_3be1_4608_a549_2bf0edf7674a.slice/crio-099a0c8977b758b67003013295847ed0eafad8a502e589365fea6b0dcea3d8c5 WatchSource:0}: Error finding container 099a0c8977b758b67003013295847ed0eafad8a502e589365fea6b0dcea3d8c5: Status 404 returned error can't find the container with id 099a0c8977b758b67003013295847ed0eafad8a502e589365fea6b0dcea3d8c5
Sep 09 15:40:09.605570 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:09.605534    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8a43b2-kxk6d" event={"ID":"fdd81934-3be1-4608-a549-2bf0edf7674a","Type":"ContainerStarted","Data":"099a0c8977b758b67003013295847ed0eafad8a502e589365fea6b0dcea3d8c5"}
Sep 09 15:40:10.612566 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:10.612472    2553 generic.go:356] "Generic (PLEG): container finished" podID="fdd81934-3be1-4608-a549-2bf0edf7674a" containerID="4b615a2a1859f58ba1901f651a7efe5547be330094e5da096162f9c59df577b8" exitCode=0
Sep 09 15:40:10.612951 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:10.612558    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8a43b2-kxk6d" event={"ID":"fdd81934-3be1-4608-a549-2bf0edf7674a","Type":"ContainerDied","Data":"4b615a2a1859f58ba1901f651a7efe5547be330094e5da096162f9c59df577b8"}
Sep 09 15:40:11.746648 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:11.746621    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8a43b2-kxk6d"
Sep 09 15:40:11.822561 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:11.822531    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdd81934-3be1-4608-a549-2bf0edf7674a-maas-api-service-ca\") pod \"fdd81934-3be1-4608-a549-2bf0edf7674a\" (UID: \"fdd81934-3be1-4608-a549-2bf0edf7674a\") "
Sep 09 15:40:11.822881 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:11.822856    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdd81934-3be1-4608-a549-2bf0edf7674a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fdd81934-3be1-4608-a549-2bf0edf7674a" (UID: "fdd81934-3be1-4608-a549-2bf0edf7674a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:40:11.923997 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:11.923965    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdd81934-3be1-4608-a549-2bf0edf7674a-maas-api-service-ca\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:40:12.623623 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:12.623596    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8a43b2-kxk6d"
Sep 09 15:40:12.623623 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:12.623595    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8a43b2-kxk6d" event={"ID":"fdd81934-3be1-4608-a549-2bf0edf7674a","Type":"ContainerDied","Data":"099a0c8977b758b67003013295847ed0eafad8a502e589365fea6b0dcea3d8c5"}
Sep 09 15:40:12.623887 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:12.623644    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="099a0c8977b758b67003013295847ed0eafad8a502e589365fea6b0dcea3d8c5"
Sep 09 15:40:14.581507 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:14.581475    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-c348b0d2-769b566b6f-qr9sm"
Sep 09 15:40:19.347617 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:19.347581    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-8a43b2-69d656764f-f7m26"]
Sep 09 15:40:19.348051 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:19.347864    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-8a43b2-69d656764f-f7m26" podUID="125b5413-0dd4-459a-b430-e6ab4e57476f" containerName="payload-processing" containerID="cri-o://a181d0ce84a81286c8cf216eee7653b477e99f3bb4c492770b274c3ec924d71e" gracePeriod=30
Sep 09 15:40:19.508358 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:19.508323    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8a43b2-kxk6d"]
Sep 09 15:40:19.513929 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:19.513902    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8a43b2-kxk6d"]
Sep 09 15:40:19.619100 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:19.619075    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-8a43b2-69d656764f-f7m26"
Sep 09 15:40:19.663115 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:19.663081    2553 generic.go:356] "Generic (PLEG): container finished" podID="125b5413-0dd4-459a-b430-e6ab4e57476f" containerID="a181d0ce84a81286c8cf216eee7653b477e99f3bb4c492770b274c3ec924d71e" exitCode=0
Sep 09 15:40:19.663285 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:19.663128    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-8a43b2-69d656764f-f7m26"
Sep 09 15:40:19.663285 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:19.663170    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-8a43b2-69d656764f-f7m26" event={"ID":"125b5413-0dd4-459a-b430-e6ab4e57476f","Type":"ContainerDied","Data":"a181d0ce84a81286c8cf216eee7653b477e99f3bb4c492770b274c3ec924d71e"}
Sep 09 15:40:19.663285 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:19.663229    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-8a43b2-69d656764f-f7m26" event={"ID":"125b5413-0dd4-459a-b430-e6ab4e57476f","Type":"ContainerDied","Data":"137d1d282c3e94c8352dde1efb773d93c54829bda3c5a0d10f7cc121b8d870c4"}
Sep 09 15:40:19.663285 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:19.663258    2553 scope.go:117] "RemoveContainer" containerID="a181d0ce84a81286c8cf216eee7653b477e99f3bb4c492770b274c3ec924d71e"
Sep 09 15:40:19.672357 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:19.672337    2553 scope.go:117] "RemoveContainer" containerID="a181d0ce84a81286c8cf216eee7653b477e99f3bb4c492770b274c3ec924d71e"
Sep 09 15:40:19.672656 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:40:19.672635    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a181d0ce84a81286c8cf216eee7653b477e99f3bb4c492770b274c3ec924d71e\": container with ID starting with a181d0ce84a81286c8cf216eee7653b477e99f3bb4c492770b274c3ec924d71e not found: ID does not exist" containerID="a181d0ce84a81286c8cf216eee7653b477e99f3bb4c492770b274c3ec924d71e"
Sep 09 15:40:19.672757 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:19.672667    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a181d0ce84a81286c8cf216eee7653b477e99f3bb4c492770b274c3ec924d71e"} err="failed to get container status \"a181d0ce84a81286c8cf216eee7653b477e99f3bb4c492770b274c3ec924d71e\": rpc error: code = NotFound desc = could not find container \"a181d0ce84a81286c8cf216eee7653b477e99f3bb4c492770b274c3ec924d71e\": container with ID starting with a181d0ce84a81286c8cf216eee7653b477e99f3bb4c492770b274c3ec924d71e not found: ID does not exist"
Sep 09 15:40:19.693317 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:19.693283    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/125b5413-0dd4-459a-b430-e6ab4e57476f-plugins-config-volume\") pod \"125b5413-0dd4-459a-b430-e6ab4e57476f\" (UID: \"125b5413-0dd4-459a-b430-e6ab4e57476f\") "
Sep 09 15:40:19.693489 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:19.693352    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtgl6\" (UniqueName: \"kubernetes.io/projected/125b5413-0dd4-459a-b430-e6ab4e57476f-kube-api-access-jtgl6\") pod \"125b5413-0dd4-459a-b430-e6ab4e57476f\" (UID: \"125b5413-0dd4-459a-b430-e6ab4e57476f\") "
Sep 09 15:40:19.693672 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:19.693647    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/125b5413-0dd4-459a-b430-e6ab4e57476f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "125b5413-0dd4-459a-b430-e6ab4e57476f" (UID: "125b5413-0dd4-459a-b430-e6ab4e57476f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:40:19.695540 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:19.695512    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/125b5413-0dd4-459a-b430-e6ab4e57476f-kube-api-access-jtgl6" (OuterVolumeSpecName: "kube-api-access-jtgl6") pod "125b5413-0dd4-459a-b430-e6ab4e57476f" (UID: "125b5413-0dd4-459a-b430-e6ab4e57476f"). InnerVolumeSpecName "kube-api-access-jtgl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:40:19.794601 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:19.794563    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtgl6\" (UniqueName: \"kubernetes.io/projected/125b5413-0dd4-459a-b430-e6ab4e57476f-kube-api-access-jtgl6\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:40:19.794601 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:19.794594    2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/125b5413-0dd4-459a-b430-e6ab4e57476f-plugins-config-volume\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:40:19.983945 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:19.983913    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-8a43b2-69d656764f-f7m26"]
Sep 09 15:40:19.987911 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:19.987874    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-8a43b2-69d656764f-f7m26"]
Sep 09 15:40:20.962609 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:20.962574    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="125b5413-0dd4-459a-b430-e6ab4e57476f" path="/var/lib/kubelet/pods/125b5413-0dd4-459a-b430-e6ab4e57476f/volumes"
Sep 09 15:40:20.963022 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:20.962943    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdd81934-3be1-4608-a549-2bf0edf7674a" path="/var/lib/kubelet/pods/fdd81934-3be1-4608-a549-2bf0edf7674a/volumes"
Sep 09 15:40:25.406580 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:25.406547    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c348b0d2-wzhgz"]
Sep 09 15:40:25.407011 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:25.406942    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="125b5413-0dd4-459a-b430-e6ab4e57476f" containerName="payload-processing"
Sep 09 15:40:25.407011 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:25.406955    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="125b5413-0dd4-459a-b430-e6ab4e57476f" containerName="payload-processing"
Sep 09 15:40:25.407011 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:25.406979    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdd81934-3be1-4608-a549-2bf0edf7674a" containerName="revoke-keys"
Sep 09 15:40:25.407011 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:25.406984    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdd81934-3be1-4608-a549-2bf0edf7674a" containerName="revoke-keys"
Sep 09 15:40:25.407159 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:25.407039    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="125b5413-0dd4-459a-b430-e6ab4e57476f" containerName="payload-processing"
Sep 09 15:40:25.407159 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:25.407048    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdd81934-3be1-4608-a549-2bf0edf7674a" containerName="revoke-keys"
Sep 09 15:40:25.411297 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:25.411279    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c348b0d2-wzhgz"
Sep 09 15:40:25.414061 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:25.414035    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2zlsf\""
Sep 09 15:40:25.414310 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:25.414094    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 15:40:25.417476 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:25.417448    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c348b0d2-wzhgz"]
Sep 09 15:40:25.449654 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:25.449619    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14d24d9b-24fa-4abf-8522-fb2978bcd369-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c348b0d2-wzhgz\" (UID: \"14d24d9b-24fa-4abf-8522-fb2978bcd369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c348b0d2-wzhgz"
Sep 09 15:40:25.550897 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:25.550863    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14d24d9b-24fa-4abf-8522-fb2978bcd369-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c348b0d2-wzhgz\" (UID: \"14d24d9b-24fa-4abf-8522-fb2978bcd369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c348b0d2-wzhgz"
Sep 09 15:40:25.551502 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:25.551478    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14d24d9b-24fa-4abf-8522-fb2978bcd369-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c348b0d2-wzhgz\" (UID: \"14d24d9b-24fa-4abf-8522-fb2978bcd369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c348b0d2-wzhgz"
Sep 09 15:40:25.722815 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:25.722779    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c348b0d2-wzhgz"
Sep 09 15:40:25.993997 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:25.993969    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c348b0d2-wzhgz"]
Sep 09 15:40:25.996130 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:40:25.996098    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d24d9b_24fa_4abf_8522_fb2978bcd369.slice/crio-205dad2bb8090ea395997a7cb98cbca6f91cef78905a87b33c9693140cf23f00 WatchSource:0}: Error finding container 205dad2bb8090ea395997a7cb98cbca6f91cef78905a87b33c9693140cf23f00: Status 404 returned error can't find the container with id 205dad2bb8090ea395997a7cb98cbca6f91cef78905a87b33c9693140cf23f00
Sep 09 15:40:26.702866 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:26.702828    2553 generic.go:356] "Generic (PLEG): container finished" podID="14d24d9b-24fa-4abf-8522-fb2978bcd369" containerID="1d94cf632fbc730193ba53ab27a4699fc904bb66222a5f8b4e521ccc53ec5e29" exitCode=0
Sep 09 15:40:26.703269 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:26.702918    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c348b0d2-wzhgz" event={"ID":"14d24d9b-24fa-4abf-8522-fb2978bcd369","Type":"ContainerDied","Data":"1d94cf632fbc730193ba53ab27a4699fc904bb66222a5f8b4e521ccc53ec5e29"}
Sep 09 15:40:26.703269 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:26.702972    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c348b0d2-wzhgz" event={"ID":"14d24d9b-24fa-4abf-8522-fb2978bcd369","Type":"ContainerStarted","Data":"205dad2bb8090ea395997a7cb98cbca6f91cef78905a87b33c9693140cf23f00"}
Sep 09 15:40:27.833859 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:27.833832    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c348b0d2-wzhgz"
Sep 09 15:40:27.873840 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:27.873792    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14d24d9b-24fa-4abf-8522-fb2978bcd369-maas-api-service-ca\") pod \"14d24d9b-24fa-4abf-8522-fb2978bcd369\" (UID: \"14d24d9b-24fa-4abf-8522-fb2978bcd369\") "
Sep 09 15:40:27.874462 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:27.874430    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14d24d9b-24fa-4abf-8522-fb2978bcd369-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "14d24d9b-24fa-4abf-8522-fb2978bcd369" (UID: "14d24d9b-24fa-4abf-8522-fb2978bcd369"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:40:27.974872 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:27.974778    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14d24d9b-24fa-4abf-8522-fb2978bcd369-maas-api-service-ca\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:40:28.715726 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:28.715692    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c348b0d2-wzhgz"
Sep 09 15:40:28.715917 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:28.715751    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c348b0d2-wzhgz" event={"ID":"14d24d9b-24fa-4abf-8522-fb2978bcd369","Type":"ContainerDied","Data":"205dad2bb8090ea395997a7cb98cbca6f91cef78905a87b33c9693140cf23f00"}
Sep 09 15:40:28.715917 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:28.715800    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="205dad2bb8090ea395997a7cb98cbca6f91cef78905a87b33c9693140cf23f00"
Sep 09 15:40:32.231069 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:32.231033    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c348b0d2-769b566b6f-qr9sm"]
Sep 09 15:40:32.231491 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:32.231317    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-c348b0d2-769b566b6f-qr9sm" podUID="679e994b-fb9a-41ca-9d21-38d38d656db9" containerName="payload-processing" containerID="cri-o://9935d35841d72f104e299fc3e4c44514d690765043a0a38d9a6ec3de27b7097d" gracePeriod=30
Sep 09 15:40:32.393115 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:32.393066    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c348b0d2-wzhgz"]
Sep 09 15:40:32.396533 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:32.396499    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c348b0d2-wzhgz"]
Sep 09 15:40:32.495479 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:32.495390    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c348b0d2-769b566b6f-qr9sm"
Sep 09 15:40:32.620215 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:32.620181    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4svms\" (UniqueName: \"kubernetes.io/projected/679e994b-fb9a-41ca-9d21-38d38d656db9-kube-api-access-4svms\") pod \"679e994b-fb9a-41ca-9d21-38d38d656db9\" (UID: \"679e994b-fb9a-41ca-9d21-38d38d656db9\") "
Sep 09 15:40:32.620385 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:32.620223    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/679e994b-fb9a-41ca-9d21-38d38d656db9-plugins-config-volume\") pod \"679e994b-fb9a-41ca-9d21-38d38d656db9\" (UID: \"679e994b-fb9a-41ca-9d21-38d38d656db9\") "
Sep 09 15:40:32.620707 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:32.620674    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/679e994b-fb9a-41ca-9d21-38d38d656db9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "679e994b-fb9a-41ca-9d21-38d38d656db9" (UID: "679e994b-fb9a-41ca-9d21-38d38d656db9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:40:32.622386 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:32.622358    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/679e994b-fb9a-41ca-9d21-38d38d656db9-kube-api-access-4svms" (OuterVolumeSpecName: "kube-api-access-4svms") pod "679e994b-fb9a-41ca-9d21-38d38d656db9" (UID: "679e994b-fb9a-41ca-9d21-38d38d656db9"). InnerVolumeSpecName "kube-api-access-4svms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:40:32.721553 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:32.721520    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4svms\" (UniqueName: \"kubernetes.io/projected/679e994b-fb9a-41ca-9d21-38d38d656db9-kube-api-access-4svms\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:40:32.721553 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:32.721548    2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/679e994b-fb9a-41ca-9d21-38d38d656db9-plugins-config-volume\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:40:32.739163 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:32.739121    2553 generic.go:356] "Generic (PLEG): container finished" podID="679e994b-fb9a-41ca-9d21-38d38d656db9" containerID="9935d35841d72f104e299fc3e4c44514d690765043a0a38d9a6ec3de27b7097d" exitCode=0
Sep 09 15:40:32.739303 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:32.739176    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c348b0d2-769b566b6f-qr9sm"
Sep 09 15:40:32.739303 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:32.739204    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c348b0d2-769b566b6f-qr9sm" event={"ID":"679e994b-fb9a-41ca-9d21-38d38d656db9","Type":"ContainerDied","Data":"9935d35841d72f104e299fc3e4c44514d690765043a0a38d9a6ec3de27b7097d"}
Sep 09 15:40:32.739303 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:32.739259    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c348b0d2-769b566b6f-qr9sm" event={"ID":"679e994b-fb9a-41ca-9d21-38d38d656db9","Type":"ContainerDied","Data":"71bc934aee7244eb399929a163068e3c91ed2a5e08517a1d9e4e8a20dcbaa482"}
Sep 09 15:40:32.739303 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:32.739283    2553 scope.go:117] "RemoveContainer" containerID="9935d35841d72f104e299fc3e4c44514d690765043a0a38d9a6ec3de27b7097d"
Sep 09 15:40:32.756135 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:32.756113    2553 scope.go:117] "RemoveContainer" containerID="9935d35841d72f104e299fc3e4c44514d690765043a0a38d9a6ec3de27b7097d"
Sep 09 15:40:32.756478 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:40:32.756451    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9935d35841d72f104e299fc3e4c44514d690765043a0a38d9a6ec3de27b7097d\": container with ID starting with 9935d35841d72f104e299fc3e4c44514d690765043a0a38d9a6ec3de27b7097d not found: ID does not exist" containerID="9935d35841d72f104e299fc3e4c44514d690765043a0a38d9a6ec3de27b7097d"
Sep 09 15:40:32.756592 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:32.756484    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9935d35841d72f104e299fc3e4c44514d690765043a0a38d9a6ec3de27b7097d"} err="failed to get container status \"9935d35841d72f104e299fc3e4c44514d690765043a0a38d9a6ec3de27b7097d\": rpc error: code = NotFound desc = could not find container \"9935d35841d72f104e299fc3e4c44514d690765043a0a38d9a6ec3de27b7097d\": container with ID starting with 9935d35841d72f104e299fc3e4c44514d690765043a0a38d9a6ec3de27b7097d not found: ID does not exist"
Sep 09 15:40:32.765593 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:32.765567    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c348b0d2-769b566b6f-qr9sm"]
Sep 09 15:40:32.769341 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:32.769316    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c348b0d2-769b566b6f-qr9sm"]
Sep 09 15:40:32.962267 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:32.962222    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14d24d9b-24fa-4abf-8522-fb2978bcd369" path="/var/lib/kubelet/pods/14d24d9b-24fa-4abf-8522-fb2978bcd369/volumes"
Sep 09 15:40:32.962687 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:40:32.962668    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="679e994b-fb9a-41ca-9d21-38d38d656db9" path="/var/lib/kubelet/pods/679e994b-fb9a-41ca-9d21-38d38d656db9/volumes"
Sep 09 15:42:08.033676 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.033639    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"]
Sep 09 15:42:08.034351 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.034023    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="679e994b-fb9a-41ca-9d21-38d38d656db9" containerName="payload-processing"
Sep 09 15:42:08.034351 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.034035    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="679e994b-fb9a-41ca-9d21-38d38d656db9" containerName="payload-processing"
Sep 09 15:42:08.034351 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.034046    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14d24d9b-24fa-4abf-8522-fb2978bcd369" containerName="revoke-keys"
Sep 09 15:42:08.034351 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.034051    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="14d24d9b-24fa-4abf-8522-fb2978bcd369" containerName="revoke-keys"
Sep 09 15:42:08.034351 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.034124    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="14d24d9b-24fa-4abf-8522-fb2978bcd369" containerName="revoke-keys"
Sep 09 15:42:08.034351 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.034135    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="679e994b-fb9a-41ca-9d21-38d38d656db9" containerName="payload-processing"
Sep 09 15:42:08.036138 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.036115    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"
Sep 09 15:42:08.038654 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.038628    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-73d87788-openshift-default-dockercfg-658b4\""
Sep 09 15:42:08.049240 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.049210    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"]
Sep 09 15:42:08.112004 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.111953    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8vz6\" (UniqueName: \"kubernetes.io/projected/e4726a71-f397-492a-b847-121dfb5f1bb5-kube-api-access-d8vz6\") pod \"e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k\" (UID: \"e4726a71-f397-492a-b847-121dfb5f1bb5\") " pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"
Sep 09 15:42:08.112181 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.112037    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4726a71-f397-492a-b847-121dfb5f1bb5-workload-certs\") pod \"e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k\" (UID: \"e4726a71-f397-492a-b847-121dfb5f1bb5\") " pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"
Sep 09 15:42:08.112181 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.112070    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4726a71-f397-492a-b847-121dfb5f1bb5-istio-podinfo\") pod \"e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k\" (UID: \"e4726a71-f397-492a-b847-121dfb5f1bb5\") " pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"
Sep 09 15:42:08.112181 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.112101    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4726a71-f397-492a-b847-121dfb5f1bb5-istiod-ca-cert\") pod \"e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k\" (UID: \"e4726a71-f397-492a-b847-121dfb5f1bb5\") " pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"
Sep 09 15:42:08.112289 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.112217    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4726a71-f397-492a-b847-121dfb5f1bb5-workload-socket\") pod \"e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k\" (UID: \"e4726a71-f397-492a-b847-121dfb5f1bb5\") " pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"
Sep 09 15:42:08.112289 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.112251    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4726a71-f397-492a-b847-121dfb5f1bb5-istio-data\") pod \"e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k\" (UID: \"e4726a71-f397-492a-b847-121dfb5f1bb5\") " pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"
Sep 09 15:42:08.112289 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.112267    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4726a71-f397-492a-b847-121dfb5f1bb5-istio-token\") pod \"e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k\" (UID: \"e4726a71-f397-492a-b847-121dfb5f1bb5\") " pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"
Sep 09 15:42:08.112381 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.112370    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4726a71-f397-492a-b847-121dfb5f1bb5-credential-socket\") pod \"e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k\" (UID: \"e4726a71-f397-492a-b847-121dfb5f1bb5\") " pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"
Sep 09 15:42:08.112414 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.112387    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4726a71-f397-492a-b847-121dfb5f1bb5-istio-envoy\") pod \"e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k\" (UID: \"e4726a71-f397-492a-b847-121dfb5f1bb5\") " pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"
Sep 09 15:42:08.213867 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.213827    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4726a71-f397-492a-b847-121dfb5f1bb5-workload-socket\") pod \"e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k\" (UID: \"e4726a71-f397-492a-b847-121dfb5f1bb5\") " pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"
Sep 09 15:42:08.213867 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.213865    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4726a71-f397-492a-b847-121dfb5f1bb5-istio-data\") pod \"e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k\" (UID: \"e4726a71-f397-492a-b847-121dfb5f1bb5\") " pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"
Sep 09 15:42:08.214104 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.213883    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4726a71-f397-492a-b847-121dfb5f1bb5-istio-token\") pod \"e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k\" (UID: \"e4726a71-f397-492a-b847-121dfb5f1bb5\") " pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"
Sep 09 15:42:08.214104 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.214042    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4726a71-f397-492a-b847-121dfb5f1bb5-credential-socket\") pod \"e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k\" (UID: \"e4726a71-f397-492a-b847-121dfb5f1bb5\") " pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"
Sep 09 15:42:08.214104 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.214088    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4726a71-f397-492a-b847-121dfb5f1bb5-istio-envoy\") pod \"e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k\" (UID: \"e4726a71-f397-492a-b847-121dfb5f1bb5\") " pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"
Sep 09 15:42:08.214243 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.214140    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8vz6\" (UniqueName: \"kubernetes.io/projected/e4726a71-f397-492a-b847-121dfb5f1bb5-kube-api-access-d8vz6\") pod \"e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k\" (UID: \"e4726a71-f397-492a-b847-121dfb5f1bb5\") " pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"
Sep 09 15:42:08.214243 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.214193    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4726a71-f397-492a-b847-121dfb5f1bb5-workload-certs\") pod \"e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k\" (UID: \"e4726a71-f397-492a-b847-121dfb5f1bb5\") " pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"
Sep 09 15:42:08.214243 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.214217    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4726a71-f397-492a-b847-121dfb5f1bb5-istio-podinfo\") pod \"e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k\" (UID: \"e4726a71-f397-492a-b847-121dfb5f1bb5\") " pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"
Sep 09 15:42:08.214385 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.214247    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4726a71-f397-492a-b847-121dfb5f1bb5-istiod-ca-cert\") pod \"e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k\" (UID: \"e4726a71-f397-492a-b847-121dfb5f1bb5\") " pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"
Sep 09 15:42:08.214385 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.214296    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4726a71-f397-492a-b847-121dfb5f1bb5-workload-socket\") pod \"e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k\" (UID: \"e4726a71-f397-492a-b847-121dfb5f1bb5\") " pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"
Sep 09 15:42:08.214385 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.214337    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4726a71-f397-492a-b847-121dfb5f1bb5-istio-data\") pod \"e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k\" (UID: \"e4726a71-f397-492a-b847-121dfb5f1bb5\") " pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"
Sep 09 15:42:08.214552 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.214424    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4726a71-f397-492a-b847-121dfb5f1bb5-credential-socket\") pod \"e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k\" (UID: \"e4726a71-f397-492a-b847-121dfb5f1bb5\") " pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"
Sep 09 15:42:08.214552 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.214525    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4726a71-f397-492a-b847-121dfb5f1bb5-workload-certs\") pod \"e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k\" (UID: \"e4726a71-f397-492a-b847-121dfb5f1bb5\") " pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"
Sep 09 15:42:08.214957 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.214937    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4726a71-f397-492a-b847-121dfb5f1bb5-istiod-ca-cert\") pod \"e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k\" (UID: \"e4726a71-f397-492a-b847-121dfb5f1bb5\") " pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"
Sep 09 15:42:08.216353 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.216332    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4726a71-f397-492a-b847-121dfb5f1bb5-istio-envoy\") pod \"e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k\" (UID: \"e4726a71-f397-492a-b847-121dfb5f1bb5\") " pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"
Sep 09 15:42:08.216661 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.216638    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4726a71-f397-492a-b847-121dfb5f1bb5-istio-podinfo\") pod \"e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k\" (UID: \"e4726a71-f397-492a-b847-121dfb5f1bb5\") " pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"
Sep 09 15:42:08.221643 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.221614    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4726a71-f397-492a-b847-121dfb5f1bb5-istio-token\") pod \"e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k\" (UID: \"e4726a71-f397-492a-b847-121dfb5f1bb5\") " pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"
Sep 09 15:42:08.221643 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.221636    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8vz6\" (UniqueName: \"kubernetes.io/projected/e4726a71-f397-492a-b847-121dfb5f1bb5-kube-api-access-d8vz6\") pod \"e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k\" (UID: \"e4726a71-f397-492a-b847-121dfb5f1bb5\") " pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"
Sep 09 15:42:08.350234 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.350140    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"
Sep 09 15:42:08.683691 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:42:08.683656    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4726a71_f397_492a_b847_121dfb5f1bb5.slice/crio-b8ea46155c41a3f7990d39538119033e35ca45abe93f581e38bb46837ca61a05 WatchSource:0}: Error finding container b8ea46155c41a3f7990d39538119033e35ca45abe93f581e38bb46837ca61a05: Status 404 returned error can't find the container with id b8ea46155c41a3f7990d39538119033e35ca45abe93f581e38bb46837ca61a05
Sep 09 15:42:08.684967 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.684943    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"]
Sep 09 15:42:08.685993 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.685957    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:42:08.686099 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.686024    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:42:08.686099 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:08.686052    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:42:09.236374 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:09.236338    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k" event={"ID":"e4726a71-f397-492a-b847-121dfb5f1bb5","Type":"ContainerStarted","Data":"19e80297a640d47ce51edc3c97a2c821e14e94ddd191e220a8f2de1692d23673"}
Sep 09 15:42:09.236830 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:09.236394    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k" event={"ID":"e4726a71-f397-492a-b847-121dfb5f1bb5","Type":"ContainerStarted","Data":"b8ea46155c41a3f7990d39538119033e35ca45abe93f581e38bb46837ca61a05"}
Sep 09 15:42:09.257981 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:09.257929    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k" podStartSLOduration=1.257912317 podStartE2EDuration="1.257912317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:42:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:42:09.257651976 +0000 UTC m=+1294.856546624" watchObservedRunningTime="2026-09-09 15:42:09.257912317 +0000 UTC m=+1294.856806980"
Sep 09 15:42:09.350511 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:09.350476    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"
Sep 09 15:42:09.356234 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:09.356205    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"
Sep 09 15:42:10.241256 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:10.241220    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"
Sep 09 15:42:10.242392 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:10.242372    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"
Sep 09 15:42:15.944205 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:15.944169    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-73d87788-859456f585-nr8st"]
Sep 09 15:42:15.947007 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:15.946985    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-73d87788-859456f585-nr8st"
Sep 09 15:42:15.950700 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:15.950675    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-73d87788\""
Sep 09 15:42:15.950910 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:15.950890    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-73d87788-dockercfg-xdv2q\""
Sep 09 15:42:15.960481 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:15.960456    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-73d87788-859456f585-nr8st"]
Sep 09 15:42:16.089939 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:16.089895    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2d026f6-40b9-47e7-b829-37e8e7296d88-plugins-config-volume\") pod \"payload-processing-e2e-mt-73d87788-859456f585-nr8st\" (UID: \"f2d026f6-40b9-47e7-b829-37e8e7296d88\") " pod="openshift-ingress/payload-processing-e2e-mt-73d87788-859456f585-nr8st"
Sep 09 15:42:16.090117 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:16.089955    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcxgc\" (UniqueName: \"kubernetes.io/projected/f2d026f6-40b9-47e7-b829-37e8e7296d88-kube-api-access-kcxgc\") pod \"payload-processing-e2e-mt-73d87788-859456f585-nr8st\" (UID: \"f2d026f6-40b9-47e7-b829-37e8e7296d88\") " pod="openshift-ingress/payload-processing-e2e-mt-73d87788-859456f585-nr8st"
Sep 09 15:42:16.191357 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:16.191317    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2d026f6-40b9-47e7-b829-37e8e7296d88-plugins-config-volume\") pod \"payload-processing-e2e-mt-73d87788-859456f585-nr8st\" (UID: \"f2d026f6-40b9-47e7-b829-37e8e7296d88\") " pod="openshift-ingress/payload-processing-e2e-mt-73d87788-859456f585-nr8st"
Sep 09 15:42:16.191526 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:16.191378    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcxgc\" (UniqueName: \"kubernetes.io/projected/f2d026f6-40b9-47e7-b829-37e8e7296d88-kube-api-access-kcxgc\") pod \"payload-processing-e2e-mt-73d87788-859456f585-nr8st\" (UID: \"f2d026f6-40b9-47e7-b829-37e8e7296d88\") " pod="openshift-ingress/payload-processing-e2e-mt-73d87788-859456f585-nr8st"
Sep 09 15:42:16.192070 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:16.192049    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2d026f6-40b9-47e7-b829-37e8e7296d88-plugins-config-volume\") pod \"payload-processing-e2e-mt-73d87788-859456f585-nr8st\" (UID: \"f2d026f6-40b9-47e7-b829-37e8e7296d88\") " pod="openshift-ingress/payload-processing-e2e-mt-73d87788-859456f585-nr8st"
Sep 09 15:42:16.200371 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:16.200300    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcxgc\" (UniqueName: \"kubernetes.io/projected/f2d026f6-40b9-47e7-b829-37e8e7296d88-kube-api-access-kcxgc\") pod \"payload-processing-e2e-mt-73d87788-859456f585-nr8st\" (UID: \"f2d026f6-40b9-47e7-b829-37e8e7296d88\") " pod="openshift-ingress/payload-processing-e2e-mt-73d87788-859456f585-nr8st"
Sep 09 15:42:16.259033 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:16.258978    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-73d87788-859456f585-nr8st"
Sep 09 15:42:16.530327 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:16.530304    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-73d87788-859456f585-nr8st"]
Sep 09 15:42:16.532709 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:42:16.532679    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2d026f6_40b9_47e7_b829_37e8e7296d88.slice/crio-d3fdfca29f46db12da9d7b455e5441cde4e4df6822434d61bbf0526a6c81b22f WatchSource:0}: Error finding container d3fdfca29f46db12da9d7b455e5441cde4e4df6822434d61bbf0526a6c81b22f: Status 404 returned error can't find the container with id d3fdfca29f46db12da9d7b455e5441cde4e4df6822434d61bbf0526a6c81b22f
Sep 09 15:42:17.281253 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:17.281215    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-73d87788-859456f585-nr8st" event={"ID":"f2d026f6-40b9-47e7-b829-37e8e7296d88","Type":"ContainerStarted","Data":"a081fc7af17400d980eed35c7da0c63599ae4c3f46ca48e94ac8c8ee1ceb836b"}
Sep 09 15:42:17.281646 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:17.281276    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-73d87788-859456f585-nr8st" event={"ID":"f2d026f6-40b9-47e7-b829-37e8e7296d88","Type":"ContainerStarted","Data":"d3fdfca29f46db12da9d7b455e5441cde4e4df6822434d61bbf0526a6c81b22f"}
Sep 09 15:42:17.281646 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:17.281330    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-73d87788-859456f585-nr8st"
Sep 09 15:42:17.297572 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:17.297520    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-73d87788-859456f585-nr8st" podStartSLOduration=2.297504563 podStartE2EDuration="2.297504563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:42:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:42:17.297211262 +0000 UTC m=+1302.896105909" watchObservedRunningTime="2026-09-09 15:42:17.297504563 +0000 UTC m=+1302.896399249"
Sep 09 15:42:20.138271 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:20.138232    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-d36ac1/explicit-model-d36ac1-kserve-6585dcd757-clmtr"]
Sep 09 15:42:20.142097 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:20.142076    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-d36ac1/explicit-model-d36ac1-kserve-6585dcd757-clmtr"
Sep 09 15:42:20.145917 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:20.145895    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-d36ac1\"/\"openshift-service-ca.crt\""
Sep 09 15:42:20.146240 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:20.145910    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-d36ac1\"/\"explicit-model-d36ac1-kserve-self-signed-certs\""
Sep 09 15:42:20.146371 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:20.145913    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-d36ac1\"/\"default-dockercfg-q9qw9\""
Sep 09 15:42:20.146506 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:20.145928    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-d36ac1\"/\"kube-root-ca.crt\""
Sep 09 15:42:20.149870 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:20.149636    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-d36ac1/explicit-model-d36ac1-kserve-6585dcd757-clmtr"]
Sep 09 15:42:20.230804 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:20.230767    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3de743f-dd6b-4ac4-91fb-3343dc9ce536-model-cache\") pod \"explicit-model-d36ac1-kserve-6585dcd757-clmtr\" (UID: \"a3de743f-dd6b-4ac4-91fb-3343dc9ce536\") " pod="ai-tenant-e2e-explicit-d36ac1/explicit-model-d36ac1-kserve-6585dcd757-clmtr"
Sep 09 15:42:20.230804 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:20.230815    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3de743f-dd6b-4ac4-91fb-3343dc9ce536-tls-certs\") pod \"explicit-model-d36ac1-kserve-6585dcd757-clmtr\" (UID: \"a3de743f-dd6b-4ac4-91fb-3343dc9ce536\") " pod="ai-tenant-e2e-explicit-d36ac1/explicit-model-d36ac1-kserve-6585dcd757-clmtr"
Sep 09 15:42:20.231015 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:20.230952    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3de743f-dd6b-4ac4-91fb-3343dc9ce536-dshm\") pod \"explicit-model-d36ac1-kserve-6585dcd757-clmtr\" (UID: \"a3de743f-dd6b-4ac4-91fb-3343dc9ce536\") " pod="ai-tenant-e2e-explicit-d36ac1/explicit-model-d36ac1-kserve-6585dcd757-clmtr"
Sep 09 15:42:20.231015 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:20.230978    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3de743f-dd6b-4ac4-91fb-3343dc9ce536-tmp-dir\") pod \"explicit-model-d36ac1-kserve-6585dcd757-clmtr\" (UID: \"a3de743f-dd6b-4ac4-91fb-3343dc9ce536\") " pod="ai-tenant-e2e-explicit-d36ac1/explicit-model-d36ac1-kserve-6585dcd757-clmtr"
Sep 09 15:42:20.231015 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:20.230997    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjmcf\" (UniqueName: \"kubernetes.io/projected/a3de743f-dd6b-4ac4-91fb-3343dc9ce536-kube-api-access-sjmcf\") pod \"explicit-model-d36ac1-kserve-6585dcd757-clmtr\" (UID: \"a3de743f-dd6b-4ac4-91fb-3343dc9ce536\") " pod="ai-tenant-e2e-explicit-d36ac1/explicit-model-d36ac1-kserve-6585dcd757-clmtr"
Sep 09 15:42:20.231113 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:20.231059    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3de743f-dd6b-4ac4-91fb-3343dc9ce536-home\") pod \"explicit-model-d36ac1-kserve-6585dcd757-clmtr\" (UID: \"a3de743f-dd6b-4ac4-91fb-3343dc9ce536\") " pod="ai-tenant-e2e-explicit-d36ac1/explicit-model-d36ac1-kserve-6585dcd757-clmtr"
Sep 09 15:42:20.331423 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:20.331391    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3de743f-dd6b-4ac4-91fb-3343dc9ce536-tmp-dir\") pod \"explicit-model-d36ac1-kserve-6585dcd757-clmtr\" (UID: \"a3de743f-dd6b-4ac4-91fb-3343dc9ce536\") " pod="ai-tenant-e2e-explicit-d36ac1/explicit-model-d36ac1-kserve-6585dcd757-clmtr"
Sep 09 15:42:20.331423 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:20.331428    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjmcf\" (UniqueName: \"kubernetes.io/projected/a3de743f-dd6b-4ac4-91fb-3343dc9ce536-kube-api-access-sjmcf\") pod \"explicit-model-d36ac1-kserve-6585dcd757-clmtr\" (UID: \"a3de743f-dd6b-4ac4-91fb-3343dc9ce536\") " pod="ai-tenant-e2e-explicit-d36ac1/explicit-model-d36ac1-kserve-6585dcd757-clmtr"
Sep 09 15:42:20.331676 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:20.331548    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3de743f-dd6b-4ac4-91fb-3343dc9ce536-home\") pod \"explicit-model-d36ac1-kserve-6585dcd757-clmtr\" (UID: \"a3de743f-dd6b-4ac4-91fb-3343dc9ce536\") " pod="ai-tenant-e2e-explicit-d36ac1/explicit-model-d36ac1-kserve-6585dcd757-clmtr"
Sep 09 15:42:20.331676 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:20.331592    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3de743f-dd6b-4ac4-91fb-3343dc9ce536-model-cache\") pod \"explicit-model-d36ac1-kserve-6585dcd757-clmtr\" (UID: \"a3de743f-dd6b-4ac4-91fb-3343dc9ce536\") " pod="ai-tenant-e2e-explicit-d36ac1/explicit-model-d36ac1-kserve-6585dcd757-clmtr"
Sep 09 15:42:20.331676 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:20.331611    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3de743f-dd6b-4ac4-91fb-3343dc9ce536-tls-certs\") pod \"explicit-model-d36ac1-kserve-6585dcd757-clmtr\" (UID: \"a3de743f-dd6b-4ac4-91fb-3343dc9ce536\") " pod="ai-tenant-e2e-explicit-d36ac1/explicit-model-d36ac1-kserve-6585dcd757-clmtr"
Sep 09 15:42:20.331864 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:20.331759    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3de743f-dd6b-4ac4-91fb-3343dc9ce536-dshm\") pod \"explicit-model-d36ac1-kserve-6585dcd757-clmtr\" (UID: \"a3de743f-dd6b-4ac4-91fb-3343dc9ce536\") " pod="ai-tenant-e2e-explicit-d36ac1/explicit-model-d36ac1-kserve-6585dcd757-clmtr"
Sep 09 15:42:20.331864 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:20.331834    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3de743f-dd6b-4ac4-91fb-3343dc9ce536-tmp-dir\") pod \"explicit-model-d36ac1-kserve-6585dcd757-clmtr\" (UID: \"a3de743f-dd6b-4ac4-91fb-3343dc9ce536\") " pod="ai-tenant-e2e-explicit-d36ac1/explicit-model-d36ac1-kserve-6585dcd757-clmtr"
Sep 09 15:42:20.331974 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:20.331912    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3de743f-dd6b-4ac4-91fb-3343dc9ce536-model-cache\") pod \"explicit-model-d36ac1-kserve-6585dcd757-clmtr\" (UID: \"a3de743f-dd6b-4ac4-91fb-3343dc9ce536\") " pod="ai-tenant-e2e-explicit-d36ac1/explicit-model-d36ac1-kserve-6585dcd757-clmtr"
Sep 09 15:42:20.332030 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:20.331984    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3de743f-dd6b-4ac4-91fb-3343dc9ce536-home\") pod \"explicit-model-d36ac1-kserve-6585dcd757-clmtr\" (UID: \"a3de743f-dd6b-4ac4-91fb-3343dc9ce536\") " pod="ai-tenant-e2e-explicit-d36ac1/explicit-model-d36ac1-kserve-6585dcd757-clmtr"
Sep 09 15:42:20.333965 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:20.333944    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3de743f-dd6b-4ac4-91fb-3343dc9ce536-dshm\") pod \"explicit-model-d36ac1-kserve-6585dcd757-clmtr\" (UID: \"a3de743f-dd6b-4ac4-91fb-3343dc9ce536\") " pod="ai-tenant-e2e-explicit-d36ac1/explicit-model-d36ac1-kserve-6585dcd757-clmtr"
Sep 09 15:42:20.334136 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:20.334117    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3de743f-dd6b-4ac4-91fb-3343dc9ce536-tls-certs\") pod \"explicit-model-d36ac1-kserve-6585dcd757-clmtr\" (UID: \"a3de743f-dd6b-4ac4-91fb-3343dc9ce536\") " pod="ai-tenant-e2e-explicit-d36ac1/explicit-model-d36ac1-kserve-6585dcd757-clmtr"
Sep 09 15:42:20.340897 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:20.340873    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjmcf\" (UniqueName: \"kubernetes.io/projected/a3de743f-dd6b-4ac4-91fb-3343dc9ce536-kube-api-access-sjmcf\") pod \"explicit-model-d36ac1-kserve-6585dcd757-clmtr\" (UID: \"a3de743f-dd6b-4ac4-91fb-3343dc9ce536\") " pod="ai-tenant-e2e-explicit-d36ac1/explicit-model-d36ac1-kserve-6585dcd757-clmtr"
Sep 09 15:42:20.452809 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:20.452775    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-d36ac1/explicit-model-d36ac1-kserve-6585dcd757-clmtr"
Sep 09 15:42:20.723852 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:20.723825    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-d36ac1/explicit-model-d36ac1-kserve-6585dcd757-clmtr"]
Sep 09 15:42:20.726122 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:42:20.726088    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3de743f_dd6b_4ac4_91fb_3343dc9ce536.slice/crio-13c1eece5f3a2354a23ff51625f5ae14797f4c71cd2d59a99d83b3d26f0d7803 WatchSource:0}: Error finding container 13c1eece5f3a2354a23ff51625f5ae14797f4c71cd2d59a99d83b3d26f0d7803: Status 404 returned error can't find the container with id 13c1eece5f3a2354a23ff51625f5ae14797f4c71cd2d59a99d83b3d26f0d7803
Sep 09 15:42:21.304545 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:21.304509    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-d36ac1/explicit-model-d36ac1-kserve-6585dcd757-clmtr" event={"ID":"a3de743f-dd6b-4ac4-91fb-3343dc9ce536","Type":"ContainerStarted","Data":"e19ce37294c275f1d154e73e278fd133d6b2d5051e512f83083df8c629645753"}
Sep 09 15:42:21.304969 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:21.304558    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-d36ac1/explicit-model-d36ac1-kserve-6585dcd757-clmtr" event={"ID":"a3de743f-dd6b-4ac4-91fb-3343dc9ce536","Type":"ContainerStarted","Data":"13c1eece5f3a2354a23ff51625f5ae14797f4c71cd2d59a99d83b3d26f0d7803"}
Sep 09 15:42:21.322019 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:21.321965    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-d36ac1/explicit-model-d36ac1-kserve-6585dcd757-clmtr" podStartSLOduration=1.3219486040000001 podStartE2EDuration="1.321948604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:42:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:42:21.320393613 +0000 UTC m=+1306.919288257" watchObservedRunningTime="2026-09-09 15:42:21.321948604 +0000 UTC m=+1306.920843247"
Sep 09 15:42:28.289389 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:28.289356    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-73d87788-859456f585-nr8st"
Sep 09 15:42:30.453272 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:30.453229    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-d36ac1/explicit-model-d36ac1-kserve-6585dcd757-clmtr"
Sep 09 15:42:30.453862 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:30.453844    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-d36ac1/explicit-model-d36ac1-kserve-6585dcd757-clmtr"
Sep 09 15:42:30.473146 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:30.473118    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-d36ac1/explicit-model-d36ac1-kserve-6585dcd757-clmtr"
Sep 09 15:42:31.381606 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:31.381576    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-d36ac1/explicit-model-d36ac1-kserve-6585dcd757-clmtr"
Sep 09 15:42:38.408717 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:38.408683    2553 generic.go:356] "Generic (PLEG): container finished" podID="e4726a71-f397-492a-b847-121dfb5f1bb5" containerID="19e80297a640d47ce51edc3c97a2c821e14e94ddd191e220a8f2de1692d23673" exitCode=0
Sep 09 15:42:38.409222 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:38.408763    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k" event={"ID":"e4726a71-f397-492a-b847-121dfb5f1bb5","Type":"ContainerDied","Data":"19e80297a640d47ce51edc3c97a2c821e14e94ddd191e220a8f2de1692d23673"}
Sep 09 15:42:38.409222 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:38.409166    2553 scope.go:117] "RemoveContainer" containerID="19e80297a640d47ce51edc3c97a2c821e14e94ddd191e220a8f2de1692d23673"
Sep 09 15:42:38.410370 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:38.410331    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:42:38.410479 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:38.410405    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:42:38.410479 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:38.410439    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:42:39.416002 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:39.415969    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k" event={"ID":"e4726a71-f397-492a-b847-121dfb5f1bb5","Type":"ContainerStarted","Data":"a8a702b170c6f4edf9a8359bb1e32bdd13ac299e50995c1d0c25d762249f1edf"}
Sep 09 15:42:40.241461 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:40.241408    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"
Sep 09 15:42:40.352054 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:40.352016    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"
Sep 09 15:42:40.352321 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:40.352050    2553 patch_prober.go:28] interesting pod/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 09 15:42:40.352487 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:40.352465    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k" podUID="e4726a71-f397-492a-b847-121dfb5f1bb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 09 15:42:41.351132 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:41.351082    2553 patch_prober.go:28] interesting pod/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 09 15:42:41.351645 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:41.351166    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k" podUID="e4726a71-f397-492a-b847-121dfb5f1bb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 09 15:42:42.351041 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:42.351000    2553 patch_prober.go:28] interesting pod/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 09 15:42:42.351386 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:42.351346    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k" podUID="e4726a71-f397-492a-b847-121dfb5f1bb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 09 15:42:43.351467 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:43.351414    2553 patch_prober.go:28] interesting pod/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 09 15:42:43.352000 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:43.351493    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k" podUID="e4726a71-f397-492a-b847-121dfb5f1bb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 09 15:42:44.351041 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:44.350999    2553 patch_prober.go:28] interesting pod/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 09 15:42:44.351361 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:44.351321    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k" podUID="e4726a71-f397-492a-b847-121dfb5f1bb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 09 15:42:45.350625 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:45.350587    2553 patch_prober.go:28] interesting pod/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 09 15:42:45.351054 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:45.350671    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k" podUID="e4726a71-f397-492a-b847-121dfb5f1bb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 09 15:42:46.230935 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:46.230897    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-73d87788-55ftt"]
Sep 09 15:42:46.233874 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:46.233848    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-73d87788-55ftt"
Sep 09 15:42:46.236619 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:46.236591    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 15:42:46.236817 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:46.236713    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2zlsf\""
Sep 09 15:42:46.243010 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:46.242979    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-73d87788-55ftt"]
Sep 09 15:42:46.359921 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:46.359876    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"
Sep 09 15:42:46.361942 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:46.361914    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"
Sep 09 15:42:46.397525 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:46.397482    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fab5ac5e-744d-4ab1-8c6a-66475eca1363-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-73d87788-55ftt\" (UID: \"fab5ac5e-744d-4ab1-8c6a-66475eca1363\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-73d87788-55ftt"
Sep 09 15:42:46.498471 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:46.498372    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fab5ac5e-744d-4ab1-8c6a-66475eca1363-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-73d87788-55ftt\" (UID: \"fab5ac5e-744d-4ab1-8c6a-66475eca1363\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-73d87788-55ftt"
Sep 09 15:42:46.499084 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:46.499061    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fab5ac5e-744d-4ab1-8c6a-66475eca1363-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-73d87788-55ftt\" (UID: \"fab5ac5e-744d-4ab1-8c6a-66475eca1363\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-73d87788-55ftt"
Sep 09 15:42:46.550434 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:46.550393    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-73d87788-55ftt"
Sep 09 15:42:46.820502 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:46.820477    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-73d87788-55ftt"]
Sep 09 15:42:46.822619 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:42:46.822592    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfab5ac5e_744d_4ab1_8c6a_66475eca1363.slice/crio-aeeec6b4f5a59866fe3e79ba1151f5f5974d8b5389709412730ce59f3b13068e WatchSource:0}: Error finding container aeeec6b4f5a59866fe3e79ba1151f5f5974d8b5389709412730ce59f3b13068e: Status 404 returned error can't find the container with id aeeec6b4f5a59866fe3e79ba1151f5f5974d8b5389709412730ce59f3b13068e
Sep 09 15:42:47.473231 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:47.473198    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-73d87788-55ftt" event={"ID":"fab5ac5e-744d-4ab1-8c6a-66475eca1363","Type":"ContainerStarted","Data":"5f556c512476fc2562c6c6df1add36b72bb3d469e2feabe9e2294d6212ceee16"}
Sep 09 15:42:47.473645 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:47.473248    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-73d87788-55ftt" event={"ID":"fab5ac5e-744d-4ab1-8c6a-66475eca1363","Type":"ContainerStarted","Data":"aeeec6b4f5a59866fe3e79ba1151f5f5974d8b5389709412730ce59f3b13068e"}
Sep 09 15:42:47.488667 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:47.488608    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-73d87788-55ftt" podStartSLOduration=1.488589775 podStartE2EDuration="1.488589775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:42:47.486843964 +0000 UTC m=+1333.085738608" watchObservedRunningTime="2026-09-09 15:42:47.488589775 +0000 UTC m=+1333.087484469"
Sep 09 15:42:48.480230 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:48.480199    2553 generic.go:356] "Generic (PLEG): container finished" podID="fab5ac5e-744d-4ab1-8c6a-66475eca1363" containerID="5f556c512476fc2562c6c6df1add36b72bb3d469e2feabe9e2294d6212ceee16" exitCode=0
Sep 09 15:42:48.480667 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:48.480267    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-73d87788-55ftt" event={"ID":"fab5ac5e-744d-4ab1-8c6a-66475eca1363","Type":"ContainerDied","Data":"5f556c512476fc2562c6c6df1add36b72bb3d469e2feabe9e2294d6212ceee16"}
Sep 09 15:42:49.626242 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:49.626217    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-73d87788-55ftt"
Sep 09 15:42:49.730422 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:49.730392    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fab5ac5e-744d-4ab1-8c6a-66475eca1363-maas-api-service-ca\") pod \"fab5ac5e-744d-4ab1-8c6a-66475eca1363\" (UID: \"fab5ac5e-744d-4ab1-8c6a-66475eca1363\") "
Sep 09 15:42:49.730782 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:49.730721    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fab5ac5e-744d-4ab1-8c6a-66475eca1363-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fab5ac5e-744d-4ab1-8c6a-66475eca1363" (UID: "fab5ac5e-744d-4ab1-8c6a-66475eca1363"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:42:49.831888 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:49.831803    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fab5ac5e-744d-4ab1-8c6a-66475eca1363-maas-api-service-ca\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:42:50.491827 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:50.491793    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-73d87788-55ftt"
Sep 09 15:42:50.491997 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:50.491800    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-73d87788-55ftt" event={"ID":"fab5ac5e-744d-4ab1-8c6a-66475eca1363","Type":"ContainerDied","Data":"aeeec6b4f5a59866fe3e79ba1151f5f5974d8b5389709412730ce59f3b13068e"}
Sep 09 15:42:50.491997 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:50.491922    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aeeec6b4f5a59866fe3e79ba1151f5f5974d8b5389709412730ce59f3b13068e"
Sep 09 15:42:53.211277 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:53.211237    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-d36ac1-8vblv"]
Sep 09 15:42:53.211803 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:53.211617    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fab5ac5e-744d-4ab1-8c6a-66475eca1363" containerName="revoke-keys"
Sep 09 15:42:53.211803 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:53.211628    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="fab5ac5e-744d-4ab1-8c6a-66475eca1363" containerName="revoke-keys"
Sep 09 15:42:53.211803 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:53.211706    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="fab5ac5e-744d-4ab1-8c6a-66475eca1363" containerName="revoke-keys"
Sep 09 15:42:53.214881 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:53.214859    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-d36ac1-8vblv"
Sep 09 15:42:53.217516 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:53.217497    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2zlsf\""
Sep 09 15:42:53.217626 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:53.217512    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 15:42:53.221964 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:53.221942    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-d36ac1-8vblv"]
Sep 09 15:42:53.367877 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:53.367843    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97538c49-c3ea-4a77-8c01-97c9271b2309-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-d36ac1-8vblv\" (UID: \"97538c49-c3ea-4a77-8c01-97c9271b2309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-d36ac1-8vblv"
Sep 09 15:42:53.468938 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:53.468838    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97538c49-c3ea-4a77-8c01-97c9271b2309-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-d36ac1-8vblv\" (UID: \"97538c49-c3ea-4a77-8c01-97c9271b2309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-d36ac1-8vblv"
Sep 09 15:42:53.469481 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:53.469460    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97538c49-c3ea-4a77-8c01-97c9271b2309-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-d36ac1-8vblv\" (UID: \"97538c49-c3ea-4a77-8c01-97c9271b2309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-d36ac1-8vblv"
Sep 09 15:42:53.527862 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:53.527827    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-d36ac1-8vblv"
Sep 09 15:42:53.797430 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:53.797405    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-d36ac1-8vblv"]
Sep 09 15:42:53.799440 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:42:53.799411    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97538c49_c3ea_4a77_8c01_97c9271b2309.slice/crio-df1891da66b97362e53a1da3f51348b3c541312949a9899f29929717dc9e68a6 WatchSource:0}: Error finding container df1891da66b97362e53a1da3f51348b3c541312949a9899f29929717dc9e68a6: Status 404 returned error can't find the container with id df1891da66b97362e53a1da3f51348b3c541312949a9899f29929717dc9e68a6
Sep 09 15:42:54.514239 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:54.514208    2553 generic.go:356] "Generic (PLEG): container finished" podID="97538c49-c3ea-4a77-8c01-97c9271b2309" containerID="ea770044cb6acb0d9352b5913b6504bc7de72e6be21939f46197e8d75247b081" exitCode=0
Sep 09 15:42:54.514682 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:54.514296    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-d36ac1-8vblv" event={"ID":"97538c49-c3ea-4a77-8c01-97c9271b2309","Type":"ContainerDied","Data":"ea770044cb6acb0d9352b5913b6504bc7de72e6be21939f46197e8d75247b081"}
Sep 09 15:42:54.514682 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:54.514347    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-d36ac1-8vblv" event={"ID":"97538c49-c3ea-4a77-8c01-97c9271b2309","Type":"ContainerStarted","Data":"df1891da66b97362e53a1da3f51348b3c541312949a9899f29929717dc9e68a6"}
Sep 09 15:42:55.662207 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:55.662179    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-d36ac1-8vblv"
Sep 09 15:42:55.691977 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:55.691944    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97538c49-c3ea-4a77-8c01-97c9271b2309-maas-api-service-ca\") pod \"97538c49-c3ea-4a77-8c01-97c9271b2309\" (UID: \"97538c49-c3ea-4a77-8c01-97c9271b2309\") "
Sep 09 15:42:55.692290 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:55.692267    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97538c49-c3ea-4a77-8c01-97c9271b2309-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "97538c49-c3ea-4a77-8c01-97c9271b2309" (UID: "97538c49-c3ea-4a77-8c01-97c9271b2309"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:42:55.793051 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:55.792967    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97538c49-c3ea-4a77-8c01-97c9271b2309-maas-api-service-ca\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:42:56.529909 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:56.529877    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-d36ac1-8vblv" event={"ID":"97538c49-c3ea-4a77-8c01-97c9271b2309","Type":"ContainerDied","Data":"df1891da66b97362e53a1da3f51348b3c541312949a9899f29929717dc9e68a6"}
Sep 09 15:42:56.530082 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:56.529920    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-d36ac1-8vblv"
Sep 09 15:42:56.530082 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:56.529925    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df1891da66b97362e53a1da3f51348b3c541312949a9899f29929717dc9e68a6"
Sep 09 15:42:57.042393 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:57.042357    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-73d87788-859456f585-nr8st"]
Sep 09 15:42:57.042829 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:57.042607    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-73d87788-859456f585-nr8st" podUID="f2d026f6-40b9-47e7-b829-37e8e7296d88" containerName="payload-processing" containerID="cri-o://a081fc7af17400d980eed35c7da0c63599ae4c3f46ca48e94ac8c8ee1ceb836b" gracePeriod=30
Sep 09 15:42:57.294662 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:57.294576    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-73d87788-55ftt"]
Sep 09 15:42:57.303196 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:57.303161    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-73d87788-55ftt"]
Sep 09 15:42:57.320556 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:57.320518    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-73d87788-859456f585-nr8st"
Sep 09 15:42:57.409398 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:57.409362    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcxgc\" (UniqueName: \"kubernetes.io/projected/f2d026f6-40b9-47e7-b829-37e8e7296d88-kube-api-access-kcxgc\") pod \"f2d026f6-40b9-47e7-b829-37e8e7296d88\" (UID: \"f2d026f6-40b9-47e7-b829-37e8e7296d88\") "
Sep 09 15:42:57.409580 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:57.409450    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2d026f6-40b9-47e7-b829-37e8e7296d88-plugins-config-volume\") pod \"f2d026f6-40b9-47e7-b829-37e8e7296d88\" (UID: \"f2d026f6-40b9-47e7-b829-37e8e7296d88\") "
Sep 09 15:42:57.409821 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:57.409794    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2d026f6-40b9-47e7-b829-37e8e7296d88-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f2d026f6-40b9-47e7-b829-37e8e7296d88" (UID: "f2d026f6-40b9-47e7-b829-37e8e7296d88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:42:57.411428 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:57.411405    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2d026f6-40b9-47e7-b829-37e8e7296d88-kube-api-access-kcxgc" (OuterVolumeSpecName: "kube-api-access-kcxgc") pod "f2d026f6-40b9-47e7-b829-37e8e7296d88" (UID: "f2d026f6-40b9-47e7-b829-37e8e7296d88"). InnerVolumeSpecName "kube-api-access-kcxgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:42:57.510111 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:57.510077    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcxgc\" (UniqueName: \"kubernetes.io/projected/f2d026f6-40b9-47e7-b829-37e8e7296d88-kube-api-access-kcxgc\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:42:57.510111 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:57.510109    2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2d026f6-40b9-47e7-b829-37e8e7296d88-plugins-config-volume\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:42:57.536200 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:57.536163    2553 generic.go:356] "Generic (PLEG): container finished" podID="f2d026f6-40b9-47e7-b829-37e8e7296d88" containerID="a081fc7af17400d980eed35c7da0c63599ae4c3f46ca48e94ac8c8ee1ceb836b" exitCode=0
Sep 09 15:42:57.536350 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:57.536208    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-73d87788-859456f585-nr8st"
Sep 09 15:42:57.536350 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:57.536217    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-73d87788-859456f585-nr8st" event={"ID":"f2d026f6-40b9-47e7-b829-37e8e7296d88","Type":"ContainerDied","Data":"a081fc7af17400d980eed35c7da0c63599ae4c3f46ca48e94ac8c8ee1ceb836b"}
Sep 09 15:42:57.536350 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:57.536289    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-73d87788-859456f585-nr8st" event={"ID":"f2d026f6-40b9-47e7-b829-37e8e7296d88","Type":"ContainerDied","Data":"d3fdfca29f46db12da9d7b455e5441cde4e4df6822434d61bbf0526a6c81b22f"}
Sep 09 15:42:57.536350 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:57.536314    2553 scope.go:117] "RemoveContainer" containerID="a081fc7af17400d980eed35c7da0c63599ae4c3f46ca48e94ac8c8ee1ceb836b"
Sep 09 15:42:57.546147 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:57.546125    2553 scope.go:117] "RemoveContainer" containerID="a081fc7af17400d980eed35c7da0c63599ae4c3f46ca48e94ac8c8ee1ceb836b"
Sep 09 15:42:57.546468 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:42:57.546446    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a081fc7af17400d980eed35c7da0c63599ae4c3f46ca48e94ac8c8ee1ceb836b\": container with ID starting with a081fc7af17400d980eed35c7da0c63599ae4c3f46ca48e94ac8c8ee1ceb836b not found: ID does not exist" containerID="a081fc7af17400d980eed35c7da0c63599ae4c3f46ca48e94ac8c8ee1ceb836b"
Sep 09 15:42:57.546522 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:57.546476    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a081fc7af17400d980eed35c7da0c63599ae4c3f46ca48e94ac8c8ee1ceb836b"} err="failed to get container status \"a081fc7af17400d980eed35c7da0c63599ae4c3f46ca48e94ac8c8ee1ceb836b\": rpc error: code = NotFound desc = could not find container \"a081fc7af17400d980eed35c7da0c63599ae4c3f46ca48e94ac8c8ee1ceb836b\": container with ID starting with a081fc7af17400d980eed35c7da0c63599ae4c3f46ca48e94ac8c8ee1ceb836b not found: ID does not exist"
Sep 09 15:42:57.559384 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:57.559352    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-73d87788-859456f585-nr8st"]
Sep 09 15:42:57.564595 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:57.564570    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-73d87788-859456f585-nr8st"]
Sep 09 15:42:58.962460 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:58.962414    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2d026f6-40b9-47e7-b829-37e8e7296d88" path="/var/lib/kubelet/pods/f2d026f6-40b9-47e7-b829-37e8e7296d88/volumes"
Sep 09 15:42:58.962967 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:42:58.962939    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fab5ac5e-744d-4ab1-8c6a-66475eca1363" path="/var/lib/kubelet/pods/fab5ac5e-744d-4ab1-8c6a-66475eca1363/volumes"
Sep 09 15:43:03.722338 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:03.722302    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"]
Sep 09 15:43:03.722712 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:03.722559    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k" podUID="e4726a71-f397-492a-b847-121dfb5f1bb5" containerName="istio-proxy" containerID="cri-o://a8a702b170c6f4edf9a8359bb1e32bdd13ac299e50995c1d0c25d762249f1edf" gracePeriod=30
Sep 09 15:43:04.200145 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:04.200107    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-d36ac1-8vblv"]
Sep 09 15:43:04.202524 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:04.202498    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-d36ac1-8vblv"]
Sep 09 15:43:04.963096 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:04.963054    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97538c49-c3ea-4a77-8c01-97c9271b2309" path="/var/lib/kubelet/pods/97538c49-c3ea-4a77-8c01-97c9271b2309/volumes"
Sep 09 15:43:08.978022 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:08.977991    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"
Sep 09 15:43:09.023750 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.023651    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8vz6\" (UniqueName: \"kubernetes.io/projected/e4726a71-f397-492a-b847-121dfb5f1bb5-kube-api-access-d8vz6\") pod \"e4726a71-f397-492a-b847-121dfb5f1bb5\" (UID: \"e4726a71-f397-492a-b847-121dfb5f1bb5\") "
Sep 09 15:43:09.023750 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.023695    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4726a71-f397-492a-b847-121dfb5f1bb5-istio-token\") pod \"e4726a71-f397-492a-b847-121dfb5f1bb5\" (UID: \"e4726a71-f397-492a-b847-121dfb5f1bb5\") "
Sep 09 15:43:09.023982 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.023757    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4726a71-f397-492a-b847-121dfb5f1bb5-workload-socket\") pod \"e4726a71-f397-492a-b847-121dfb5f1bb5\" (UID: \"e4726a71-f397-492a-b847-121dfb5f1bb5\") "
Sep 09 15:43:09.023982 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.023792    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4726a71-f397-492a-b847-121dfb5f1bb5-credential-socket\") pod \"e4726a71-f397-492a-b847-121dfb5f1bb5\" (UID: \"e4726a71-f397-492a-b847-121dfb5f1bb5\") "
Sep 09 15:43:09.023982 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.023841    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4726a71-f397-492a-b847-121dfb5f1bb5-istiod-ca-cert\") pod \"e4726a71-f397-492a-b847-121dfb5f1bb5\" (UID: \"e4726a71-f397-492a-b847-121dfb5f1bb5\") "
Sep 09 15:43:09.023982 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.023860    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4726a71-f397-492a-b847-121dfb5f1bb5-istio-data\") pod \"e4726a71-f397-492a-b847-121dfb5f1bb5\" (UID: \"e4726a71-f397-492a-b847-121dfb5f1bb5\") "
Sep 09 15:43:09.023982 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.023881    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4726a71-f397-492a-b847-121dfb5f1bb5-workload-certs\") pod \"e4726a71-f397-492a-b847-121dfb5f1bb5\" (UID: \"e4726a71-f397-492a-b847-121dfb5f1bb5\") "
Sep 09 15:43:09.024370 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.024346    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4726a71-f397-492a-b847-121dfb5f1bb5-istio-podinfo\") pod \"e4726a71-f397-492a-b847-121dfb5f1bb5\" (UID: \"e4726a71-f397-492a-b847-121dfb5f1bb5\") "
Sep 09 15:43:09.024370 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.024351    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4726a71-f397-492a-b847-121dfb5f1bb5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e4726a71-f397-492a-b847-121dfb5f1bb5" (UID: "e4726a71-f397-492a-b847-121dfb5f1bb5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:43:09.024485 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.024401    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4726a71-f397-492a-b847-121dfb5f1bb5-istio-envoy\") pod \"e4726a71-f397-492a-b847-121dfb5f1bb5\" (UID: \"e4726a71-f397-492a-b847-121dfb5f1bb5\") "
Sep 09 15:43:09.024993 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.024949    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4726a71-f397-492a-b847-121dfb5f1bb5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e4726a71-f397-492a-b847-121dfb5f1bb5" (UID: "e4726a71-f397-492a-b847-121dfb5f1bb5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:43:09.025426 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.025403    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4726a71-f397-492a-b847-121dfb5f1bb5-istio-data" (OuterVolumeSpecName: "istio-data") pod "e4726a71-f397-492a-b847-121dfb5f1bb5" (UID: "e4726a71-f397-492a-b847-121dfb5f1bb5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:43:09.025628 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.025602    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4726a71-f397-492a-b847-121dfb5f1bb5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e4726a71-f397-492a-b847-121dfb5f1bb5" (UID: "e4726a71-f397-492a-b847-121dfb5f1bb5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:43:09.025748 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.025710    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4726a71-f397-492a-b847-121dfb5f1bb5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e4726a71-f397-492a-b847-121dfb5f1bb5" (UID: "e4726a71-f397-492a-b847-121dfb5f1bb5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:43:09.025975 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.025957    2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4726a71-f397-492a-b847-121dfb5f1bb5-workload-socket\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:43:09.026075 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.026062    2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4726a71-f397-492a-b847-121dfb5f1bb5-credential-socket\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:43:09.026159 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.026148    2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4726a71-f397-492a-b847-121dfb5f1bb5-istiod-ca-cert\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:43:09.026239 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.026230    2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4726a71-f397-492a-b847-121dfb5f1bb5-istio-data\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:43:09.026843 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.026822    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4726a71-f397-492a-b847-121dfb5f1bb5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e4726a71-f397-492a-b847-121dfb5f1bb5" (UID: "e4726a71-f397-492a-b847-121dfb5f1bb5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 15:43:09.028602 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.028580    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4726a71-f397-492a-b847-121dfb5f1bb5-kube-api-access-d8vz6" (OuterVolumeSpecName: "kube-api-access-d8vz6") pod "e4726a71-f397-492a-b847-121dfb5f1bb5" (UID: "e4726a71-f397-492a-b847-121dfb5f1bb5"). InnerVolumeSpecName "kube-api-access-d8vz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:43:09.035291 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.035071    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4726a71-f397-492a-b847-121dfb5f1bb5-istio-token" (OuterVolumeSpecName: "istio-token") pod "e4726a71-f397-492a-b847-121dfb5f1bb5" (UID: "e4726a71-f397-492a-b847-121dfb5f1bb5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:43:09.036111 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.036084    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4726a71-f397-492a-b847-121dfb5f1bb5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e4726a71-f397-492a-b847-121dfb5f1bb5" (UID: "e4726a71-f397-492a-b847-121dfb5f1bb5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:43:09.127056 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.127015    2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4726a71-f397-492a-b847-121dfb5f1bb5-workload-certs\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:43:09.127056 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.127051    2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4726a71-f397-492a-b847-121dfb5f1bb5-istio-podinfo\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:43:09.127056 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.127060    2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4726a71-f397-492a-b847-121dfb5f1bb5-istio-envoy\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:43:09.127291 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.127069    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8vz6\" (UniqueName: \"kubernetes.io/projected/e4726a71-f397-492a-b847-121dfb5f1bb5-kube-api-access-d8vz6\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:43:09.127291 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.127081    2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4726a71-f397-492a-b847-121dfb5f1bb5-istio-token\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:43:09.541473 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.541430    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-d36ac1/explicit-model-d36ac1-kserve-6585dcd757-clmtr"]
Sep 09 15:43:09.604642 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.604603    2553 generic.go:356] "Generic (PLEG): container finished" podID="e4726a71-f397-492a-b847-121dfb5f1bb5" containerID="a8a702b170c6f4edf9a8359bb1e32bdd13ac299e50995c1d0c25d762249f1edf" exitCode=0
Sep 09 15:43:09.604861 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.604675    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"
Sep 09 15:43:09.604861 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.604680    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k" event={"ID":"e4726a71-f397-492a-b847-121dfb5f1bb5","Type":"ContainerDied","Data":"a8a702b170c6f4edf9a8359bb1e32bdd13ac299e50995c1d0c25d762249f1edf"}
Sep 09 15:43:09.604861 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.604784    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k" event={"ID":"e4726a71-f397-492a-b847-121dfb5f1bb5","Type":"ContainerDied","Data":"b8ea46155c41a3f7990d39538119033e35ca45abe93f581e38bb46837ca61a05"}
Sep 09 15:43:09.604861 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.604808    2553 scope.go:117] "RemoveContainer" containerID="a8a702b170c6f4edf9a8359bb1e32bdd13ac299e50995c1d0c25d762249f1edf"
Sep 09 15:43:09.614229 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.614206    2553 scope.go:117] "RemoveContainer" containerID="19e80297a640d47ce51edc3c97a2c821e14e94ddd191e220a8f2de1692d23673"
Sep 09 15:43:09.622604 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.622584    2553 scope.go:117] "RemoveContainer" containerID="a8a702b170c6f4edf9a8359bb1e32bdd13ac299e50995c1d0c25d762249f1edf"
Sep 09 15:43:09.622915 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:43:09.622892    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8a702b170c6f4edf9a8359bb1e32bdd13ac299e50995c1d0c25d762249f1edf\": container with ID starting with a8a702b170c6f4edf9a8359bb1e32bdd13ac299e50995c1d0c25d762249f1edf not found: ID does not exist" containerID="a8a702b170c6f4edf9a8359bb1e32bdd13ac299e50995c1d0c25d762249f1edf"
Sep 09 15:43:09.623012 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.622926    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8a702b170c6f4edf9a8359bb1e32bdd13ac299e50995c1d0c25d762249f1edf"} err="failed to get container status \"a8a702b170c6f4edf9a8359bb1e32bdd13ac299e50995c1d0c25d762249f1edf\": rpc error: code = NotFound desc = could not find container \"a8a702b170c6f4edf9a8359bb1e32bdd13ac299e50995c1d0c25d762249f1edf\": container with ID starting with a8a702b170c6f4edf9a8359bb1e32bdd13ac299e50995c1d0c25d762249f1edf not found: ID does not exist"
Sep 09 15:43:09.623012 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.622951    2553 scope.go:117] "RemoveContainer" containerID="19e80297a640d47ce51edc3c97a2c821e14e94ddd191e220a8f2de1692d23673"
Sep 09 15:43:09.623216 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:43:09.623199    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19e80297a640d47ce51edc3c97a2c821e14e94ddd191e220a8f2de1692d23673\": container with ID starting with 19e80297a640d47ce51edc3c97a2c821e14e94ddd191e220a8f2de1692d23673 not found: ID does not exist" containerID="19e80297a640d47ce51edc3c97a2c821e14e94ddd191e220a8f2de1692d23673"
Sep 09 15:43:09.623252 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.623221    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19e80297a640d47ce51edc3c97a2c821e14e94ddd191e220a8f2de1692d23673"} err="failed to get container status \"19e80297a640d47ce51edc3c97a2c821e14e94ddd191e220a8f2de1692d23673\": rpc error: code = NotFound desc = could not find container \"19e80297a640d47ce51edc3c97a2c821e14e94ddd191e220a8f2de1692d23673\": container with ID starting with 19e80297a640d47ce51edc3c97a2c821e14e94ddd191e220a8f2de1692d23673 not found: ID does not exist"
Sep 09 15:43:09.627501 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.627476    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"]
Sep 09 15:43:09.633775 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:09.633751    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-73d87788-openshift-default-859cd6dcbd-8g77k"]
Sep 09 15:43:10.962906 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:10.962872    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4726a71-f397-492a-b847-121dfb5f1bb5" path="/var/lib/kubelet/pods/e4726a71-f397-492a-b847-121dfb5f1bb5/volumes"
Sep 09 15:43:22.371124 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:22.371085    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce3053-7d64fc88d6-rmk9b"]
Sep 09 15:43:22.371571 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:22.371464    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4726a71-f397-492a-b847-121dfb5f1bb5" containerName="istio-proxy"
Sep 09 15:43:22.371571 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:22.371476    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4726a71-f397-492a-b847-121dfb5f1bb5" containerName="istio-proxy"
Sep 09 15:43:22.371571 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:22.371490    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97538c49-c3ea-4a77-8c01-97c9271b2309" containerName="revoke-keys"
Sep 09 15:43:22.371571 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:22.371496    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="97538c49-c3ea-4a77-8c01-97c9271b2309" containerName="revoke-keys"
Sep 09 15:43:22.371571 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:22.371502    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2d026f6-40b9-47e7-b829-37e8e7296d88" containerName="payload-processing"
Sep 09 15:43:22.371571 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:22.371509    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2d026f6-40b9-47e7-b829-37e8e7296d88" containerName="payload-processing"
Sep 09 15:43:22.371778 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:22.371584    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2d026f6-40b9-47e7-b829-37e8e7296d88" containerName="payload-processing"
Sep 09 15:43:22.371778 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:22.371591    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4726a71-f397-492a-b847-121dfb5f1bb5" containerName="istio-proxy"
Sep 09 15:43:22.371778 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:22.371597    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="97538c49-c3ea-4a77-8c01-97c9271b2309" containerName="revoke-keys"
Sep 09 15:43:22.376300 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:22.376276    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce3053-7d64fc88d6-rmk9b"
Sep 09 15:43:22.378977 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:22.378952    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-ce3053\""
Sep 09 15:43:22.379099 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:22.379003    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-ce3053-dockercfg-wc6f7\""
Sep 09 15:43:22.386709 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:22.386684    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce3053-7d64fc88d6-rmk9b"]
Sep 09 15:43:22.562529 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:22.562489    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f9f978e-3d58-4841-99f4-d1f26ff35f3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-ce3053-7d64fc88d6-rmk9b\" (UID: \"8f9f978e-3d58-4841-99f4-d1f26ff35f3a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce3053-7d64fc88d6-rmk9b"
Sep 09 15:43:22.562704 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:22.562623    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz2qg\" (UniqueName: \"kubernetes.io/projected/8f9f978e-3d58-4841-99f4-d1f26ff35f3a-kube-api-access-qz2qg\") pod \"payload-pre-processing-e2e-shared-a-w2-ce3053-7d64fc88d6-rmk9b\" (UID: \"8f9f978e-3d58-4841-99f4-d1f26ff35f3a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce3053-7d64fc88d6-rmk9b"
Sep 09 15:43:22.663720 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:22.663603    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz2qg\" (UniqueName: \"kubernetes.io/projected/8f9f978e-3d58-4841-99f4-d1f26ff35f3a-kube-api-access-qz2qg\") pod \"payload-pre-processing-e2e-shared-a-w2-ce3053-7d64fc88d6-rmk9b\" (UID: \"8f9f978e-3d58-4841-99f4-d1f26ff35f3a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce3053-7d64fc88d6-rmk9b"
Sep 09 15:43:22.663720 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:22.663668    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f9f978e-3d58-4841-99f4-d1f26ff35f3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-ce3053-7d64fc88d6-rmk9b\" (UID: \"8f9f978e-3d58-4841-99f4-d1f26ff35f3a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce3053-7d64fc88d6-rmk9b"
Sep 09 15:43:22.664307 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:22.664286    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f9f978e-3d58-4841-99f4-d1f26ff35f3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-ce3053-7d64fc88d6-rmk9b\" (UID: \"8f9f978e-3d58-4841-99f4-d1f26ff35f3a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce3053-7d64fc88d6-rmk9b"
Sep 09 15:43:22.679338 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:22.679311    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz2qg\" (UniqueName: \"kubernetes.io/projected/8f9f978e-3d58-4841-99f4-d1f26ff35f3a-kube-api-access-qz2qg\") pod \"payload-pre-processing-e2e-shared-a-w2-ce3053-7d64fc88d6-rmk9b\" (UID: \"8f9f978e-3d58-4841-99f4-d1f26ff35f3a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce3053-7d64fc88d6-rmk9b"
Sep 09 15:43:22.687198 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:22.687177    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce3053-7d64fc88d6-rmk9b"
Sep 09 15:43:22.963096 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:22.963067    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce3053-7d64fc88d6-rmk9b"]
Sep 09 15:43:22.963893 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:43:22.963868    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f9f978e_3d58_4841_99f4_d1f26ff35f3a.slice/crio-5f4a0e09ab8f2e64644508d73adf5f2924a3d3ec2494d7cf1c2a45881adc0fab WatchSource:0}: Error finding container 5f4a0e09ab8f2e64644508d73adf5f2924a3d3ec2494d7cf1c2a45881adc0fab: Status 404 returned error can't find the container with id 5f4a0e09ab8f2e64644508d73adf5f2924a3d3ec2494d7cf1c2a45881adc0fab
Sep 09 15:43:23.685068 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:23.685035    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce3053-7d64fc88d6-rmk9b" event={"ID":"8f9f978e-3d58-4841-99f4-d1f26ff35f3a","Type":"ContainerStarted","Data":"d5641858d020b5a0fb204c4b5630b2ec76aeaa23edea23a8c1a53162b237cce6"}
Sep 09 15:43:23.685485 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:23.685091    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce3053-7d64fc88d6-rmk9b" event={"ID":"8f9f978e-3d58-4841-99f4-d1f26ff35f3a","Type":"ContainerStarted","Data":"5f4a0e09ab8f2e64644508d73adf5f2924a3d3ec2494d7cf1c2a45881adc0fab"}
Sep 09 15:43:23.685485 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:23.685130    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce3053-7d64fc88d6-rmk9b"
Sep 09 15:43:23.711161 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:23.711096    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce3053-7d64fc88d6-rmk9b" podStartSLOduration=1.711076336 podStartE2EDuration="1.711076336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:43:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:43:23.709481447 +0000 UTC m=+1369.308376102" watchObservedRunningTime="2026-09-09 15:43:23.711076336 +0000 UTC m=+1369.309970982"
Sep 09 15:43:24.562031 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:24.561980    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-d36ac1/explicit-model-d36ac1-kserve-6585dcd757-clmtr" podUID="a3de743f-dd6b-4ac4-91fb-3343dc9ce536" containerName="main" containerID="cri-o://e19ce37294c275f1d154e73e278fd133d6b2d5051e512f83083df8c629645753" gracePeriod=45
Sep 09 15:43:24.694648 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:24.694617    2553 generic.go:356] "Generic (PLEG): container finished" podID="a3de743f-dd6b-4ac4-91fb-3343dc9ce536" containerID="e19ce37294c275f1d154e73e278fd133d6b2d5051e512f83083df8c629645753" exitCode=0
Sep 09 15:43:24.695028 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:24.694686    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-d36ac1/explicit-model-d36ac1-kserve-6585dcd757-clmtr" event={"ID":"a3de743f-dd6b-4ac4-91fb-3343dc9ce536","Type":"ContainerDied","Data":"e19ce37294c275f1d154e73e278fd133d6b2d5051e512f83083df8c629645753"}
Sep 09 15:43:24.816412 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:24.816342    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-d36ac1/explicit-model-d36ac1-kserve-6585dcd757-clmtr"
Sep 09 15:43:24.987221 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:24.987185    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3de743f-dd6b-4ac4-91fb-3343dc9ce536-tmp-dir\") pod \"a3de743f-dd6b-4ac4-91fb-3343dc9ce536\" (UID: \"a3de743f-dd6b-4ac4-91fb-3343dc9ce536\") "
Sep 09 15:43:24.987403 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:24.987269    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3de743f-dd6b-4ac4-91fb-3343dc9ce536-dshm\") pod \"a3de743f-dd6b-4ac4-91fb-3343dc9ce536\" (UID: \"a3de743f-dd6b-4ac4-91fb-3343dc9ce536\") "
Sep 09 15:43:24.987403 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:24.987302    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3de743f-dd6b-4ac4-91fb-3343dc9ce536-tls-certs\") pod \"a3de743f-dd6b-4ac4-91fb-3343dc9ce536\" (UID: \"a3de743f-dd6b-4ac4-91fb-3343dc9ce536\") "
Sep 09 15:43:24.987403 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:24.987343    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3de743f-dd6b-4ac4-91fb-3343dc9ce536-home\") pod \"a3de743f-dd6b-4ac4-91fb-3343dc9ce536\" (UID: \"a3de743f-dd6b-4ac4-91fb-3343dc9ce536\") "
Sep 09 15:43:24.987403 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:24.987401    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjmcf\" (UniqueName: \"kubernetes.io/projected/a3de743f-dd6b-4ac4-91fb-3343dc9ce536-kube-api-access-sjmcf\") pod \"a3de743f-dd6b-4ac4-91fb-3343dc9ce536\" (UID: \"a3de743f-dd6b-4ac4-91fb-3343dc9ce536\") "
Sep 09 15:43:24.987634 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:24.987480    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3de743f-dd6b-4ac4-91fb-3343dc9ce536-model-cache\") pod \"a3de743f-dd6b-4ac4-91fb-3343dc9ce536\" (UID: \"a3de743f-dd6b-4ac4-91fb-3343dc9ce536\") "
Sep 09 15:43:24.987634 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:24.987514    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3de743f-dd6b-4ac4-91fb-3343dc9ce536-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a3de743f-dd6b-4ac4-91fb-3343dc9ce536" (UID: "a3de743f-dd6b-4ac4-91fb-3343dc9ce536"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:43:24.987851 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:24.987827    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3de743f-dd6b-4ac4-91fb-3343dc9ce536-tmp-dir\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:43:24.987959 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:24.987819    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3de743f-dd6b-4ac4-91fb-3343dc9ce536-model-cache" (OuterVolumeSpecName: "model-cache") pod "a3de743f-dd6b-4ac4-91fb-3343dc9ce536" (UID: "a3de743f-dd6b-4ac4-91fb-3343dc9ce536"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:43:24.988024 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:24.987970    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3de743f-dd6b-4ac4-91fb-3343dc9ce536-home" (OuterVolumeSpecName: "home") pod "a3de743f-dd6b-4ac4-91fb-3343dc9ce536" (UID: "a3de743f-dd6b-4ac4-91fb-3343dc9ce536"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:43:24.989660 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:24.989639    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3de743f-dd6b-4ac4-91fb-3343dc9ce536-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a3de743f-dd6b-4ac4-91fb-3343dc9ce536" (UID: "a3de743f-dd6b-4ac4-91fb-3343dc9ce536"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:43:24.990008 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:24.989976    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3de743f-dd6b-4ac4-91fb-3343dc9ce536-kube-api-access-sjmcf" (OuterVolumeSpecName: "kube-api-access-sjmcf") pod "a3de743f-dd6b-4ac4-91fb-3343dc9ce536" (UID: "a3de743f-dd6b-4ac4-91fb-3343dc9ce536"). InnerVolumeSpecName "kube-api-access-sjmcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:43:24.990144 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:24.990115    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3de743f-dd6b-4ac4-91fb-3343dc9ce536-dshm" (OuterVolumeSpecName: "dshm") pod "a3de743f-dd6b-4ac4-91fb-3343dc9ce536" (UID: "a3de743f-dd6b-4ac4-91fb-3343dc9ce536"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:43:25.089477 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:25.089377    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3de743f-dd6b-4ac4-91fb-3343dc9ce536-dshm\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:43:25.089477 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:25.089424    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3de743f-dd6b-4ac4-91fb-3343dc9ce536-tls-certs\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:43:25.089477 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:25.089440    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3de743f-dd6b-4ac4-91fb-3343dc9ce536-home\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:43:25.089477 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:25.089454    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjmcf\" (UniqueName: \"kubernetes.io/projected/a3de743f-dd6b-4ac4-91fb-3343dc9ce536-kube-api-access-sjmcf\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:43:25.089477 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:25.089468    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3de743f-dd6b-4ac4-91fb-3343dc9ce536-model-cache\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:43:25.701344 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:25.701312    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-d36ac1/explicit-model-d36ac1-kserve-6585dcd757-clmtr" event={"ID":"a3de743f-dd6b-4ac4-91fb-3343dc9ce536","Type":"ContainerDied","Data":"13c1eece5f3a2354a23ff51625f5ae14797f4c71cd2d59a99d83b3d26f0d7803"}
Sep 09 15:43:25.701788 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:25.701366    2553 scope.go:117] "RemoveContainer" containerID="e19ce37294c275f1d154e73e278fd133d6b2d5051e512f83083df8c629645753"
Sep 09 15:43:25.701788 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:25.701319    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-d36ac1/explicit-model-d36ac1-kserve-6585dcd757-clmtr"
Sep 09 15:43:25.725323 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:25.725292    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-d36ac1/explicit-model-d36ac1-kserve-6585dcd757-clmtr"]
Sep 09 15:43:25.728352 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:25.728323    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-d36ac1/explicit-model-d36ac1-kserve-6585dcd757-clmtr"]
Sep 09 15:43:26.962235 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:26.962198    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3de743f-dd6b-4ac4-91fb-3343dc9ce536" path="/var/lib/kubelet/pods/a3de743f-dd6b-4ac4-91fb-3343dc9ce536/volumes"
Sep 09 15:43:34.697135 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:34.697102    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce3053-7d64fc88d6-rmk9b"
Sep 09 15:43:46.338951 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.338914    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"]
Sep 09 15:43:46.339483 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.339457    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4726a71-f397-492a-b847-121dfb5f1bb5" containerName="istio-proxy"
Sep 09 15:43:46.339483 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.339474    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4726a71-f397-492a-b847-121dfb5f1bb5" containerName="istio-proxy"
Sep 09 15:43:46.339625 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.339499    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3de743f-dd6b-4ac4-91fb-3343dc9ce536" containerName="main"
Sep 09 15:43:46.339625 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.339509    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3de743f-dd6b-4ac4-91fb-3343dc9ce536" containerName="main"
Sep 09 15:43:46.339625 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.339613    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3de743f-dd6b-4ac4-91fb-3343dc9ce536" containerName="main"
Sep 09 15:43:46.339625 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.339627    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4726a71-f397-492a-b847-121dfb5f1bb5" containerName="istio-proxy"
Sep 09 15:43:46.344322 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.344298    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"
Sep 09 15:43:46.347593 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.347570    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-081ed3-openshift-default-dockercfg-bvcfp\""
Sep 09 15:43:46.354908 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.354884    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"]
Sep 09 15:43:46.385857 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.385822    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-credential-socket\") pod \"e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx\" (UID: \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\") " pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"
Sep 09 15:43:46.385857 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.385857    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-istiod-ca-cert\") pod \"e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx\" (UID: \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\") " pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"
Sep 09 15:43:46.386067 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.385874    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twqlx\" (UniqueName: \"kubernetes.io/projected/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-kube-api-access-twqlx\") pod \"e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx\" (UID: \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\") " pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"
Sep 09 15:43:46.386067 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.385907    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-istio-podinfo\") pod \"e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx\" (UID: \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\") " pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"
Sep 09 15:43:46.386067 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.385993    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-workload-certs\") pod \"e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx\" (UID: \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\") " pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"
Sep 09 15:43:46.386067 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.386034    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-workload-socket\") pod \"e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx\" (UID: \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\") " pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"
Sep 09 15:43:46.386067 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.386056    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-istio-token\") pod \"e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx\" (UID: \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\") " pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"
Sep 09 15:43:46.386269 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.386078    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-istio-envoy\") pod \"e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx\" (UID: \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\") " pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"
Sep 09 15:43:46.386269 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.386180    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-istio-data\") pod \"e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx\" (UID: \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\") " pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"
Sep 09 15:43:46.487785 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.487744    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-istio-data\") pod \"e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx\" (UID: \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\") " pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"
Sep 09 15:43:46.487955 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.487814    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-credential-socket\") pod \"e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx\" (UID: \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\") " pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"
Sep 09 15:43:46.487955 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.487839    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-istiod-ca-cert\") pod \"e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx\" (UID: \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\") " pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"
Sep 09 15:43:46.487955 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.487864    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twqlx\" (UniqueName: \"kubernetes.io/projected/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-kube-api-access-twqlx\") pod \"e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx\" (UID: \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\") " pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"
Sep 09 15:43:46.487955 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.487894    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-istio-podinfo\") pod \"e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx\" (UID: \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\") " pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"
Sep 09 15:43:46.487955 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.487951    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-workload-certs\") pod \"e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx\" (UID: \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\") " pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"
Sep 09 15:43:46.488241 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.487983    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-workload-socket\") pod \"e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx\" (UID: \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\") " pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"
Sep 09 15:43:46.488241 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.488010    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-istio-token\") pod \"e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx\" (UID: \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\") " pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"
Sep 09 15:43:46.488241 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.488041    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-istio-envoy\") pod \"e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx\" (UID: \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\") " pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"
Sep 09 15:43:46.488241 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.488124    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-istio-data\") pod \"e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx\" (UID: \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\") " pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"
Sep 09 15:43:46.488241 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.488177    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-credential-socket\") pod \"e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx\" (UID: \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\") " pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"
Sep 09 15:43:46.488506 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.488475    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-workload-certs\") pod \"e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx\" (UID: \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\") " pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"
Sep 09 15:43:46.488579 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.488538    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-workload-socket\") pod \"e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx\" (UID: \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\") " pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"
Sep 09 15:43:46.488655 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.488637    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-istiod-ca-cert\") pod \"e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx\" (UID: \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\") " pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"
Sep 09 15:43:46.490451 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.490426    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-istio-envoy\") pod \"e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx\" (UID: \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\") " pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"
Sep 09 15:43:46.490581 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.490524    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-istio-podinfo\") pod \"e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx\" (UID: \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\") " pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"
Sep 09 15:43:46.497161 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.497141    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-istio-token\") pod \"e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx\" (UID: \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\") " pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"
Sep 09 15:43:46.497376 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.497354    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twqlx\" (UniqueName: \"kubernetes.io/projected/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-kube-api-access-twqlx\") pod \"e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx\" (UID: \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\") " pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"
Sep 09 15:43:46.656643 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.656543    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"
Sep 09 15:43:46.938512 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.938481    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"]
Sep 09 15:43:46.940821 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:43:46.940792    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd3a5c3d_2ed5_4f09_bddd_87fafe3c715b.slice/crio-70be6a131e0e763d9467f6b392110e01be96a8c382967f8f1368dd91eb747f35 WatchSource:0}: Error finding container 70be6a131e0e763d9467f6b392110e01be96a8c382967f8f1368dd91eb747f35: Status 404 returned error can't find the container with id 70be6a131e0e763d9467f6b392110e01be96a8c382967f8f1368dd91eb747f35
Sep 09 15:43:46.943829 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.943792    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:43:46.943923 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.943871    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:43:46.943923 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:46.943903    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:43:47.821223 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:47.821187    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx" event={"ID":"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b","Type":"ContainerStarted","Data":"e77362da4234b47af04d29e17c383cc0b6f299475e9508a68a9ddbd65f961242"}
Sep 09 15:43:47.821659 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:47.821245    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx" event={"ID":"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b","Type":"ContainerStarted","Data":"70be6a131e0e763d9467f6b392110e01be96a8c382967f8f1368dd91eb747f35"}
Sep 09 15:43:47.843491 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:47.843434    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx" podStartSLOduration=1.843415656 podStartE2EDuration="1.843415656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:43:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:43:47.841723194 +0000 UTC m=+1393.440617839" watchObservedRunningTime="2026-09-09 15:43:47.843415656 +0000 UTC m=+1393.442310363"
Sep 09 15:43:48.079997 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.079911    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"]
Sep 09 15:43:48.083348 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.083323    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"
Sep 09 15:43:48.086065 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.086042    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-a0938b-openshift-default-dockercfg-d256x\""
Sep 09 15:43:48.097688 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.097665    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"]
Sep 09 15:43:48.203814 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.203777    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5485277a-06c9-432d-ab56-dcfd6cbec788-istio-data\") pod \"e2e-noten-a0938b-openshift-default-7f7799b54b-886dm\" (UID: \"5485277a-06c9-432d-ab56-dcfd6cbec788\") " pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"
Sep 09 15:43:48.204024 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.203838    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5485277a-06c9-432d-ab56-dcfd6cbec788-istio-podinfo\") pod \"e2e-noten-a0938b-openshift-default-7f7799b54b-886dm\" (UID: \"5485277a-06c9-432d-ab56-dcfd6cbec788\") " pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"
Sep 09 15:43:48.204024 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.203953    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5485277a-06c9-432d-ab56-dcfd6cbec788-istiod-ca-cert\") pod \"e2e-noten-a0938b-openshift-default-7f7799b54b-886dm\" (UID: \"5485277a-06c9-432d-ab56-dcfd6cbec788\") " pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"
Sep 09 15:43:48.204024 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.203988    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b4gj\" (UniqueName: \"kubernetes.io/projected/5485277a-06c9-432d-ab56-dcfd6cbec788-kube-api-access-4b4gj\") pod \"e2e-noten-a0938b-openshift-default-7f7799b54b-886dm\" (UID: \"5485277a-06c9-432d-ab56-dcfd6cbec788\") " pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"
Sep 09 15:43:48.204185 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.204039    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5485277a-06c9-432d-ab56-dcfd6cbec788-istio-envoy\") pod \"e2e-noten-a0938b-openshift-default-7f7799b54b-886dm\" (UID: \"5485277a-06c9-432d-ab56-dcfd6cbec788\") " pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"
Sep 09 15:43:48.204185 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.204074    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5485277a-06c9-432d-ab56-dcfd6cbec788-credential-socket\") pod \"e2e-noten-a0938b-openshift-default-7f7799b54b-886dm\" (UID: \"5485277a-06c9-432d-ab56-dcfd6cbec788\") " pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"
Sep 09 15:43:48.204185 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.204115    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5485277a-06c9-432d-ab56-dcfd6cbec788-workload-certs\") pod \"e2e-noten-a0938b-openshift-default-7f7799b54b-886dm\" (UID: \"5485277a-06c9-432d-ab56-dcfd6cbec788\") " pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"
Sep 09 15:43:48.204185 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.204137    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5485277a-06c9-432d-ab56-dcfd6cbec788-istio-token\") pod \"e2e-noten-a0938b-openshift-default-7f7799b54b-886dm\" (UID: \"5485277a-06c9-432d-ab56-dcfd6cbec788\") " pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"
Sep 09 15:43:48.204385 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.204185    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5485277a-06c9-432d-ab56-dcfd6cbec788-workload-socket\") pod \"e2e-noten-a0938b-openshift-default-7f7799b54b-886dm\" (UID: \"5485277a-06c9-432d-ab56-dcfd6cbec788\") " pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"
Sep 09 15:43:48.305204 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.305163    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5485277a-06c9-432d-ab56-dcfd6cbec788-istio-data\") pod \"e2e-noten-a0938b-openshift-default-7f7799b54b-886dm\" (UID: \"5485277a-06c9-432d-ab56-dcfd6cbec788\") " pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"
Sep 09 15:43:48.305423 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.305228    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5485277a-06c9-432d-ab56-dcfd6cbec788-istio-podinfo\") pod \"e2e-noten-a0938b-openshift-default-7f7799b54b-886dm\" (UID: \"5485277a-06c9-432d-ab56-dcfd6cbec788\") " pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"
Sep 09 15:43:48.305423 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.305262    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5485277a-06c9-432d-ab56-dcfd6cbec788-istiod-ca-cert\") pod \"e2e-noten-a0938b-openshift-default-7f7799b54b-886dm\" (UID: \"5485277a-06c9-432d-ab56-dcfd6cbec788\") " pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"
Sep 09 15:43:48.305423 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.305278    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b4gj\" (UniqueName: \"kubernetes.io/projected/5485277a-06c9-432d-ab56-dcfd6cbec788-kube-api-access-4b4gj\") pod \"e2e-noten-a0938b-openshift-default-7f7799b54b-886dm\" (UID: \"5485277a-06c9-432d-ab56-dcfd6cbec788\") " pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"
Sep 09 15:43:48.305423 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.305308    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5485277a-06c9-432d-ab56-dcfd6cbec788-istio-envoy\") pod \"e2e-noten-a0938b-openshift-default-7f7799b54b-886dm\" (UID: \"5485277a-06c9-432d-ab56-dcfd6cbec788\") " pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"
Sep 09 15:43:48.305423 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.305341    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5485277a-06c9-432d-ab56-dcfd6cbec788-credential-socket\") pod \"e2e-noten-a0938b-openshift-default-7f7799b54b-886dm\" (UID: \"5485277a-06c9-432d-ab56-dcfd6cbec788\") " pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"
Sep 09 15:43:48.305423 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.305403    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5485277a-06c9-432d-ab56-dcfd6cbec788-workload-certs\") pod \"e2e-noten-a0938b-openshift-default-7f7799b54b-886dm\" (UID: \"5485277a-06c9-432d-ab56-dcfd6cbec788\") " pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"
Sep 09 15:43:48.305771 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.305429    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5485277a-06c9-432d-ab56-dcfd6cbec788-istio-token\") pod \"e2e-noten-a0938b-openshift-default-7f7799b54b-886dm\" (UID: \"5485277a-06c9-432d-ab56-dcfd6cbec788\") " pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"
Sep 09 15:43:48.305771 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.305471    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5485277a-06c9-432d-ab56-dcfd6cbec788-workload-socket\") pod \"e2e-noten-a0938b-openshift-default-7f7799b54b-886dm\" (UID: \"5485277a-06c9-432d-ab56-dcfd6cbec788\") " pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"
Sep 09 15:43:48.305771 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.305627    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5485277a-06c9-432d-ab56-dcfd6cbec788-istio-data\") pod \"e2e-noten-a0938b-openshift-default-7f7799b54b-886dm\" (UID: \"5485277a-06c9-432d-ab56-dcfd6cbec788\") " pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"
Sep 09 15:43:48.305922 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.305857    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5485277a-06c9-432d-ab56-dcfd6cbec788-credential-socket\") pod \"e2e-noten-a0938b-openshift-default-7f7799b54b-886dm\" (UID: \"5485277a-06c9-432d-ab56-dcfd6cbec788\") " pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"
Sep 09 15:43:48.306057 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.306035    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5485277a-06c9-432d-ab56-dcfd6cbec788-workload-certs\") pod \"e2e-noten-a0938b-openshift-default-7f7799b54b-886dm\" (UID: \"5485277a-06c9-432d-ab56-dcfd6cbec788\") " pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"
Sep 09 15:43:48.306115 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.306092    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5485277a-06c9-432d-ab56-dcfd6cbec788-istiod-ca-cert\") pod \"e2e-noten-a0938b-openshift-default-7f7799b54b-886dm\" (UID: \"5485277a-06c9-432d-ab56-dcfd6cbec788\") " pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"
Sep 09 15:43:48.306168 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.306117    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5485277a-06c9-432d-ab56-dcfd6cbec788-workload-socket\") pod \"e2e-noten-a0938b-openshift-default-7f7799b54b-886dm\" (UID: \"5485277a-06c9-432d-ab56-dcfd6cbec788\") " pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"
Sep 09 15:43:48.307741 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.307709    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5485277a-06c9-432d-ab56-dcfd6cbec788-istio-envoy\") pod \"e2e-noten-a0938b-openshift-default-7f7799b54b-886dm\" (UID: \"5485277a-06c9-432d-ab56-dcfd6cbec788\") " pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"
Sep 09 15:43:48.307839 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.307819    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5485277a-06c9-432d-ab56-dcfd6cbec788-istio-podinfo\") pod \"e2e-noten-a0938b-openshift-default-7f7799b54b-886dm\" (UID: \"5485277a-06c9-432d-ab56-dcfd6cbec788\") " pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"
Sep 09 15:43:48.313631 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.313608    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b4gj\" (UniqueName: \"kubernetes.io/projected/5485277a-06c9-432d-ab56-dcfd6cbec788-kube-api-access-4b4gj\") pod \"e2e-noten-a0938b-openshift-default-7f7799b54b-886dm\" (UID: \"5485277a-06c9-432d-ab56-dcfd6cbec788\") " pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"
Sep 09 15:43:48.313810 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.313669    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5485277a-06c9-432d-ab56-dcfd6cbec788-istio-token\") pod \"e2e-noten-a0938b-openshift-default-7f7799b54b-886dm\" (UID: \"5485277a-06c9-432d-ab56-dcfd6cbec788\") " pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"
Sep 09 15:43:48.394967 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.394871    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"
Sep 09 15:43:48.657060 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.656978    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"
Sep 09 15:43:48.662838 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.662812    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"
Sep 09 15:43:48.672695 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.672672    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"]
Sep 09 15:43:48.674847 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:43:48.674819    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5485277a_06c9_432d_ab56_dcfd6cbec788.slice/crio-8ba3e2a9bd801b8e1b63c2f02397d879b2944f7057c1f6569ddc1331718fe6b2 WatchSource:0}: Error finding container 8ba3e2a9bd801b8e1b63c2f02397d879b2944f7057c1f6569ddc1331718fe6b2: Status 404 returned error can't find the container with id 8ba3e2a9bd801b8e1b63c2f02397d879b2944f7057c1f6569ddc1331718fe6b2
Sep 09 15:43:48.678006 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.677958    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:43:48.678237 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.678122    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:43:48.678480 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.678451    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:43:48.828827 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.828790    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm" event={"ID":"5485277a-06c9-432d-ab56-dcfd6cbec788","Type":"ContainerStarted","Data":"dad9932d50f94faf06a7650f76ac883aa86d9d7c4730b16a57bb1f0d2ac34ff3"}
Sep 09 15:43:48.829235 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.828884    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm" event={"ID":"5485277a-06c9-432d-ab56-dcfd6cbec788","Type":"ContainerStarted","Data":"8ba3e2a9bd801b8e1b63c2f02397d879b2944f7057c1f6569ddc1331718fe6b2"}
Sep 09 15:43:48.829235 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.829056    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"
Sep 09 15:43:48.830080 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.830061    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"
Sep 09 15:43:48.849793 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:48.849722    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm" podStartSLOduration=0.849705509 podStartE2EDuration="849.705509ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:43:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:43:48.847317068 +0000 UTC m=+1394.446211725" watchObservedRunningTime="2026-09-09 15:43:48.849705509 +0000 UTC m=+1394.448600150"
Sep 09 15:43:49.395932 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:49.395892    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"
Sep 09 15:43:49.401073 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:49.401047    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"
Sep 09 15:43:49.834799 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:49.834767    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"
Sep 09 15:43:49.835692 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:49.835672    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"
Sep 09 15:43:52.257605 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:52.257568    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-081ed3-6b74b9d8f-f4zz9"]
Sep 09 15:43:52.261309 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:52.261288    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-081ed3-6b74b9d8f-f4zz9"
Sep 09 15:43:52.263863 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:52.263837    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-081ed3-dockercfg-xdg9l\""
Sep 09 15:43:52.263863 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:52.263849    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-081ed3\""
Sep 09 15:43:52.271607 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:52.271581    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-081ed3-6b74b9d8f-f4zz9"]
Sep 09 15:43:52.347449 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:52.347407    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1eca92b-d2aa-4f69-85df-4a67708a08a9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-081ed3-6b74b9d8f-f4zz9\" (UID: \"e1eca92b-d2aa-4f69-85df-4a67708a08a9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-081ed3-6b74b9d8f-f4zz9"
Sep 09 15:43:52.347866 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:52.347843    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7n6g\" (UniqueName: \"kubernetes.io/projected/e1eca92b-d2aa-4f69-85df-4a67708a08a9-kube-api-access-l7n6g\") pod \"payload-processing-e2e-shared-b-w2-081ed3-6b74b9d8f-f4zz9\" (UID: \"e1eca92b-d2aa-4f69-85df-4a67708a08a9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-081ed3-6b74b9d8f-f4zz9"
Sep 09 15:43:52.449239 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:52.449197    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7n6g\" (UniqueName: \"kubernetes.io/projected/e1eca92b-d2aa-4f69-85df-4a67708a08a9-kube-api-access-l7n6g\") pod \"payload-processing-e2e-shared-b-w2-081ed3-6b74b9d8f-f4zz9\" (UID: \"e1eca92b-d2aa-4f69-85df-4a67708a08a9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-081ed3-6b74b9d8f-f4zz9"
Sep 09 15:43:52.449416 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:52.449286    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1eca92b-d2aa-4f69-85df-4a67708a08a9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-081ed3-6b74b9d8f-f4zz9\" (UID: \"e1eca92b-d2aa-4f69-85df-4a67708a08a9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-081ed3-6b74b9d8f-f4zz9"
Sep 09 15:43:52.450062 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:52.450043    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1eca92b-d2aa-4f69-85df-4a67708a08a9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-081ed3-6b74b9d8f-f4zz9\" (UID: \"e1eca92b-d2aa-4f69-85df-4a67708a08a9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-081ed3-6b74b9d8f-f4zz9"
Sep 09 15:43:52.456986 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:52.456955    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7n6g\" (UniqueName: \"kubernetes.io/projected/e1eca92b-d2aa-4f69-85df-4a67708a08a9-kube-api-access-l7n6g\") pod \"payload-processing-e2e-shared-b-w2-081ed3-6b74b9d8f-f4zz9\" (UID: \"e1eca92b-d2aa-4f69-85df-4a67708a08a9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-081ed3-6b74b9d8f-f4zz9"
Sep 09 15:43:52.573255 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:52.573167    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-081ed3-6b74b9d8f-f4zz9"
Sep 09 15:43:52.845871 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:52.845793    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-081ed3-6b74b9d8f-f4zz9"]
Sep 09 15:43:52.849302 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:43:52.849277    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1eca92b_d2aa_4f69_85df_4a67708a08a9.slice/crio-fe6f3f3224620c819b9090c7169fe4974a4fd40389fde62cfd2cee0a2bf948f4 WatchSource:0}: Error finding container fe6f3f3224620c819b9090c7169fe4974a4fd40389fde62cfd2cee0a2bf948f4: Status 404 returned error can't find the container with id fe6f3f3224620c819b9090c7169fe4974a4fd40389fde62cfd2cee0a2bf948f4
Sep 09 15:43:53.860149 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:53.860116    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-081ed3-6b74b9d8f-f4zz9" event={"ID":"e1eca92b-d2aa-4f69-85df-4a67708a08a9","Type":"ContainerStarted","Data":"81aac16796328cbbe91d7b375db3465bb21907b2f207a7b67cad85f1688cac3b"}
Sep 09 15:43:53.860607 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:53.860172    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-081ed3-6b74b9d8f-f4zz9" event={"ID":"e1eca92b-d2aa-4f69-85df-4a67708a08a9","Type":"ContainerStarted","Data":"fe6f3f3224620c819b9090c7169fe4974a4fd40389fde62cfd2cee0a2bf948f4"}
Sep 09 15:43:53.860607 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:53.860243    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-081ed3-6b74b9d8f-f4zz9"
Sep 09 15:43:53.879047 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:43:53.878966    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-081ed3-6b74b9d8f-f4zz9" podStartSLOduration=1.878945265 podStartE2EDuration="1.878945265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:43:53.876414925 +0000 UTC m=+1399.475309569" watchObservedRunningTime="2026-09-09 15:43:53.878945265 +0000 UTC m=+1399.477839912"
Sep 09 15:44:04.869696 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:04.869659    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-081ed3-6b74b9d8f-f4zz9"
Sep 09 15:44:10.957514 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:10.957483    2553 generic.go:356] "Generic (PLEG): container finished" podID="cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b" containerID="e77362da4234b47af04d29e17c383cc0b6f299475e9508a68a9ddbd65f961242" exitCode=0
Sep 09 15:44:10.961677 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:10.961646    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx" event={"ID":"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b","Type":"ContainerDied","Data":"e77362da4234b47af04d29e17c383cc0b6f299475e9508a68a9ddbd65f961242"}
Sep 09 15:44:10.962016 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:10.961999    2553 scope.go:117] "RemoveContainer" containerID="e77362da4234b47af04d29e17c383cc0b6f299475e9508a68a9ddbd65f961242"
Sep 09 15:44:10.963135 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:10.963102    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:44:10.963303 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:10.963205    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:44:10.963303 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:10.963255    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:44:11.965757 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:11.965696    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx" event={"ID":"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b","Type":"ContainerStarted","Data":"18fb07461c9bc8d8ab91b77ae5af9db257101c11aeea06b47466d0e6a6292da8"}
Sep 09 15:44:12.658074 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:12.657264    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"
Sep 09 15:44:12.658074 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:12.657715    2553 patch_prober.go:28] interesting pod/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 09 15:44:12.658074 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:12.657793    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx" podUID="cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 09 15:44:13.657425 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:13.657382    2553 patch_prober.go:28] interesting pod/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 09 15:44:13.658529 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:13.658054    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx" podUID="cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 09 15:44:14.657466 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:14.657422    2553 patch_prober.go:28] interesting pod/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 09 15:44:14.658069 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:14.658035    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx" podUID="cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 09 15:44:15.604348 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:15.604314    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7460-140212187042992"]
Sep 09 15:44:15.607613 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:15.607592    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7460-140212187042992"
Sep 09 15:44:15.610248 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:15.610224    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pcz7f\""
Sep 09 15:44:15.615919 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:15.615898    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7460-140212187042992"]
Sep 09 15:44:15.658902 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:15.658847    2553 patch_prober.go:28] interesting pod/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 09 15:44:15.659515 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:15.659480    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx" podUID="cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 09 15:44:15.687197 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:15.687158    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4w9x\" (UniqueName: \"kubernetes.io/projected/91a657f1-19fc-44a0-a398-552269883c49-kube-api-access-k4w9x\") pod \"test-curl-7460-140212187042992\" (UID: \"91a657f1-19fc-44a0-a398-552269883c49\") " pod="opendatahub/test-curl-7460-140212187042992"
Sep 09 15:44:15.788027 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:15.787981    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4w9x\" (UniqueName: \"kubernetes.io/projected/91a657f1-19fc-44a0-a398-552269883c49-kube-api-access-k4w9x\") pod \"test-curl-7460-140212187042992\" (UID: \"91a657f1-19fc-44a0-a398-552269883c49\") " pod="opendatahub/test-curl-7460-140212187042992"
Sep 09 15:44:15.799757 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:15.799713    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4w9x\" (UniqueName: \"kubernetes.io/projected/91a657f1-19fc-44a0-a398-552269883c49-kube-api-access-k4w9x\") pod \"test-curl-7460-140212187042992\" (UID: \"91a657f1-19fc-44a0-a398-552269883c49\") " pod="opendatahub/test-curl-7460-140212187042992"
Sep 09 15:44:15.918011 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:15.917923    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7460-140212187042992"
Sep 09 15:44:16.211859 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:16.211813    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7460-140212187042992"]
Sep 09 15:44:16.214962 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:44:16.214934    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91a657f1_19fc_44a0_a398_552269883c49.slice/crio-4a1e39b6de3093488ed8c1665ed0013faf6b3c6e462aee52684e0cd6cc1024d8 WatchSource:0}: Error finding container 4a1e39b6de3093488ed8c1665ed0013faf6b3c6e462aee52684e0cd6cc1024d8: Status 404 returned error can't find the container with id 4a1e39b6de3093488ed8c1665ed0013faf6b3c6e462aee52684e0cd6cc1024d8
Sep 09 15:44:16.216276 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:16.216257    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 15:44:16.660265 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:16.660166    2553 patch_prober.go:28] interesting pod/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 09 15:44:16.660883 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:16.660844    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx" podUID="cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 09 15:44:16.999728 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:16.999671    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7460-140212187042992" event={"ID":"91a657f1-19fc-44a0-a398-552269883c49","Type":"ContainerStarted","Data":"4a1e39b6de3093488ed8c1665ed0013faf6b3c6e462aee52684e0cd6cc1024d8"}
Sep 09 15:44:17.821462 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:17.821423    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"
Sep 09 15:44:17.822199 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:17.822165    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"
Sep 09 15:44:17.823118 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:17.823098    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"
Sep 09 15:44:19.013678 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:19.013644    2553 generic.go:356] "Generic (PLEG): container finished" podID="91a657f1-19fc-44a0-a398-552269883c49" containerID="ae359b1b857fefc0b52bf06ecccbe7e1c96bc9674ce792f244226d0911672608" exitCode=0
Sep 09 15:44:19.014094 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:19.013701    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7460-140212187042992" event={"ID":"91a657f1-19fc-44a0-a398-552269883c49","Type":"ContainerDied","Data":"ae359b1b857fefc0b52bf06ecccbe7e1c96bc9674ce792f244226d0911672608"}
Sep 09 15:44:20.165528 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:20.165502    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7460-140212187042992"
Sep 09 15:44:20.208491 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:20.208453    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7460-140212187042992_91a657f1-19fc-44a0-a398-552269883c49/test-curl-7460-140212187042992/0.log"
Sep 09 15:44:20.233243 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:20.233210    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4w9x\" (UniqueName: \"kubernetes.io/projected/91a657f1-19fc-44a0-a398-552269883c49-kube-api-access-k4w9x\") pod \"91a657f1-19fc-44a0-a398-552269883c49\" (UID: \"91a657f1-19fc-44a0-a398-552269883c49\") "
Sep 09 15:44:20.234244 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:20.234219    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7460-140212187042992"]
Sep 09 15:44:20.235379 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:20.235352    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91a657f1-19fc-44a0-a398-552269883c49-kube-api-access-k4w9x" (OuterVolumeSpecName: "kube-api-access-k4w9x") pod "91a657f1-19fc-44a0-a398-552269883c49" (UID: "91a657f1-19fc-44a0-a398-552269883c49"). InnerVolumeSpecName "kube-api-access-k4w9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:44:20.239441 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:20.239418    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7460-140212187042992"]
Sep 09 15:44:20.334986 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:20.334898    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4w9x\" (UniqueName: \"kubernetes.io/projected/91a657f1-19fc-44a0-a398-552269883c49-kube-api-access-k4w9x\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:44:20.367939 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:20.367903    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7460-140212187042992"]
Sep 09 15:44:20.368320 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:20.368307    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91a657f1-19fc-44a0-a398-552269883c49" containerName="test-curl-7460-140212187042992"
Sep 09 15:44:20.368364 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:20.368321    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="91a657f1-19fc-44a0-a398-552269883c49" containerName="test-curl-7460-140212187042992"
Sep 09 15:44:20.368399 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:20.368378    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="91a657f1-19fc-44a0-a398-552269883c49" containerName="test-curl-7460-140212187042992"
Sep 09 15:44:20.371380 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:20.371362    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7460-140212187042992"
Sep 09 15:44:20.377712 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:20.377682    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7460-140212187042992"]
Sep 09 15:44:20.436361 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:20.436326    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksm2t\" (UniqueName: \"kubernetes.io/projected/bcb08f4f-e467-4fef-99f3-bc57ba984ac0-kube-api-access-ksm2t\") pod \"test-curl-7460-140212187042992\" (UID: \"bcb08f4f-e467-4fef-99f3-bc57ba984ac0\") " pod="opendatahub/test-curl-7460-140212187042992"
Sep 09 15:44:20.536984 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:20.536940    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksm2t\" (UniqueName: \"kubernetes.io/projected/bcb08f4f-e467-4fef-99f3-bc57ba984ac0-kube-api-access-ksm2t\") pod \"test-curl-7460-140212187042992\" (UID: \"bcb08f4f-e467-4fef-99f3-bc57ba984ac0\") " pod="opendatahub/test-curl-7460-140212187042992"
Sep 09 15:44:20.549458 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:20.549430    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksm2t\" (UniqueName: \"kubernetes.io/projected/bcb08f4f-e467-4fef-99f3-bc57ba984ac0-kube-api-access-ksm2t\") pod \"test-curl-7460-140212187042992\" (UID: \"bcb08f4f-e467-4fef-99f3-bc57ba984ac0\") " pod="opendatahub/test-curl-7460-140212187042992"
Sep 09 15:44:20.683435 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:20.683401    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7460-140212187042992"
Sep 09 15:44:20.959557 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:44:20.959531    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbcb08f4f_e467_4fef_99f3_bc57ba984ac0.slice/crio-cef41c8b6a9252f08d954b3204ba2a19d58e40f23c9a615df6da1a210f073636 WatchSource:0}: Error finding container cef41c8b6a9252f08d954b3204ba2a19d58e40f23c9a615df6da1a210f073636: Status 404 returned error can't find the container with id cef41c8b6a9252f08d954b3204ba2a19d58e40f23c9a615df6da1a210f073636
Sep 09 15:44:20.965510 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:20.965484    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91a657f1-19fc-44a0-a398-552269883c49" path="/var/lib/kubelet/pods/91a657f1-19fc-44a0-a398-552269883c49/volumes"
Sep 09 15:44:20.965897 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:20.965883    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7460-140212187042992"]
Sep 09 15:44:21.025565 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:21.025535    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7460-140212187042992"
Sep 09 15:44:21.025804 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:21.025585    2553 scope.go:117] "RemoveContainer" containerID="ae359b1b857fefc0b52bf06ecccbe7e1c96bc9674ce792f244226d0911672608"
Sep 09 15:44:21.027037 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:21.027011    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7460-140212187042992" event={"ID":"bcb08f4f-e467-4fef-99f3-bc57ba984ac0","Type":"ContainerStarted","Data":"cef41c8b6a9252f08d954b3204ba2a19d58e40f23c9a615df6da1a210f073636"}
Sep 09 15:44:22.033424 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:22.033389    2553 generic.go:356] "Generic (PLEG): container finished" podID="bcb08f4f-e467-4fef-99f3-bc57ba984ac0" containerID="be1ea39a3a4f07ee6caf1825e187ea1c31c3a1e87524f169209560418e930e5d" exitCode=0
Sep 09 15:44:22.033889 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:22.033480    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7460-140212187042992" event={"ID":"bcb08f4f-e467-4fef-99f3-bc57ba984ac0","Type":"ContainerDied","Data":"be1ea39a3a4f07ee6caf1825e187ea1c31c3a1e87524f169209560418e930e5d"}
Sep 09 15:44:23.177433 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:23.177409    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7460-140212187042992"
Sep 09 15:44:23.198676 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:23.198639    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7460-140212187042992_bcb08f4f-e467-4fef-99f3-bc57ba984ac0/test-curl-7460-140212187042992/0.log"
Sep 09 15:44:23.222030 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:23.221994    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7460-140212187042992"]
Sep 09 15:44:23.225985 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:23.225958    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7460-140212187042992"]
Sep 09 15:44:23.266593 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:23.266558    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ksm2t\" (UniqueName: \"kubernetes.io/projected/bcb08f4f-e467-4fef-99f3-bc57ba984ac0-kube-api-access-ksm2t\") pod \"bcb08f4f-e467-4fef-99f3-bc57ba984ac0\" (UID: \"bcb08f4f-e467-4fef-99f3-bc57ba984ac0\") "
Sep 09 15:44:23.268689 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:23.268661    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcb08f4f-e467-4fef-99f3-bc57ba984ac0-kube-api-access-ksm2t" (OuterVolumeSpecName: "kube-api-access-ksm2t") pod "bcb08f4f-e467-4fef-99f3-bc57ba984ac0" (UID: "bcb08f4f-e467-4fef-99f3-bc57ba984ac0"). InnerVolumeSpecName "kube-api-access-ksm2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:44:23.367926 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:23.367842    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksm2t\" (UniqueName: \"kubernetes.io/projected/bcb08f4f-e467-4fef-99f3-bc57ba984ac0-kube-api-access-ksm2t\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:44:23.369464 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:23.369441    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7460-140212187043568"]
Sep 09 15:44:23.369847 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:23.369832    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcb08f4f-e467-4fef-99f3-bc57ba984ac0" containerName="test-curl-7460-140212187042992"
Sep 09 15:44:23.369896 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:23.369848    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcb08f4f-e467-4fef-99f3-bc57ba984ac0" containerName="test-curl-7460-140212187042992"
Sep 09 15:44:23.369931 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:23.369911    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcb08f4f-e467-4fef-99f3-bc57ba984ac0" containerName="test-curl-7460-140212187042992"
Sep 09 15:44:23.372944 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:23.372930    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7460-140212187043568"
Sep 09 15:44:23.379631 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:23.379606    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7460-140212187043568"]
Sep 09 15:44:23.468558 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:23.468518    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs6wj\" (UniqueName: \"kubernetes.io/projected/d1ca0ea9-f1f1-48d7-befd-ec1d96fae29b-kube-api-access-hs6wj\") pod \"test-curl-7460-140212187043568\" (UID: \"d1ca0ea9-f1f1-48d7-befd-ec1d96fae29b\") " pod="opendatahub/test-curl-7460-140212187043568"
Sep 09 15:44:23.570028 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:23.569985    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs6wj\" (UniqueName: \"kubernetes.io/projected/d1ca0ea9-f1f1-48d7-befd-ec1d96fae29b-kube-api-access-hs6wj\") pod \"test-curl-7460-140212187043568\" (UID: \"d1ca0ea9-f1f1-48d7-befd-ec1d96fae29b\") " pod="opendatahub/test-curl-7460-140212187043568"
Sep 09 15:44:23.578027 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:23.578002    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs6wj\" (UniqueName: \"kubernetes.io/projected/d1ca0ea9-f1f1-48d7-befd-ec1d96fae29b-kube-api-access-hs6wj\") pod \"test-curl-7460-140212187043568\" (UID: \"d1ca0ea9-f1f1-48d7-befd-ec1d96fae29b\") " pod="opendatahub/test-curl-7460-140212187043568"
Sep 09 15:44:23.683611 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:23.683575    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7460-140212187043568"
Sep 09 15:44:23.954839 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:23.954811    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7460-140212187043568"]
Sep 09 15:44:23.956601 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:44:23.956574    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1ca0ea9_f1f1_48d7_befd_ec1d96fae29b.slice/crio-aa68888bfa787393fd228347fb6919a7c7a2b60a7e059db78b26b811b04e4e30 WatchSource:0}: Error finding container aa68888bfa787393fd228347fb6919a7c7a2b60a7e059db78b26b811b04e4e30: Status 404 returned error can't find the container with id aa68888bfa787393fd228347fb6919a7c7a2b60a7e059db78b26b811b04e4e30
Sep 09 15:44:24.047655 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:24.047627    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7460-140212187042992"
Sep 09 15:44:24.047880 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:24.047631    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cef41c8b6a9252f08d954b3204ba2a19d58e40f23c9a615df6da1a210f073636"
Sep 09 15:44:24.048978 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:24.048947    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7460-140212187043568" event={"ID":"d1ca0ea9-f1f1-48d7-befd-ec1d96fae29b","Type":"ContainerStarted","Data":"aa68888bfa787393fd228347fb6919a7c7a2b60a7e059db78b26b811b04e4e30"}
Sep 09 15:44:24.058779 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:44:24.058727    2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7460-140212187042992\" is forbidden: User \"system:node:ip-10-0-129-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-170.ec2.internal' and this object" podUID="bcb08f4f-e467-4fef-99f3-bc57ba984ac0" pod="opendatahub/test-curl-7460-140212187042992"
Sep 09 15:44:24.960820 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:44:24.960786    2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7460-140212187042992\" is forbidden: User \"system:node:ip-10-0-129-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-170.ec2.internal' and this object" podUID="bcb08f4f-e467-4fef-99f3-bc57ba984ac0" pod="opendatahub/test-curl-7460-140212187042992"
Sep 09 15:44:24.962196 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:24.962173    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcb08f4f-e467-4fef-99f3-bc57ba984ac0" path="/var/lib/kubelet/pods/bcb08f4f-e467-4fef-99f3-bc57ba984ac0/volumes"
Sep 09 15:44:25.056174 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:25.056081    2553 generic.go:356] "Generic (PLEG): container finished" podID="d1ca0ea9-f1f1-48d7-befd-ec1d96fae29b" containerID="13bd64cd179914b0e3aee253a4b8f4e83f6a732998f49f899b6c6ee6415e76b4" exitCode=0
Sep 09 15:44:25.056174 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:25.056134    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7460-140212187043568" event={"ID":"d1ca0ea9-f1f1-48d7-befd-ec1d96fae29b","Type":"ContainerDied","Data":"13bd64cd179914b0e3aee253a4b8f4e83f6a732998f49f899b6c6ee6415e76b4"}
Sep 09 15:44:26.201579 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:26.201551    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7460-140212187043568"
Sep 09 15:44:26.224460 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:26.224432    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7460-140212187043568_d1ca0ea9-f1f1-48d7-befd-ec1d96fae29b/test-curl-7460-140212187043568/0.log"
Sep 09 15:44:26.260438 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:26.260405    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7460-140212187043568"]
Sep 09 15:44:26.264269 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:26.264239    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7460-140212187043568"]
Sep 09 15:44:26.294568 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:26.294534    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hs6wj\" (UniqueName: \"kubernetes.io/projected/d1ca0ea9-f1f1-48d7-befd-ec1d96fae29b-kube-api-access-hs6wj\") pod \"d1ca0ea9-f1f1-48d7-befd-ec1d96fae29b\" (UID: \"d1ca0ea9-f1f1-48d7-befd-ec1d96fae29b\") "
Sep 09 15:44:26.296583 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:26.296554    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1ca0ea9-f1f1-48d7-befd-ec1d96fae29b-kube-api-access-hs6wj" (OuterVolumeSpecName: "kube-api-access-hs6wj") pod "d1ca0ea9-f1f1-48d7-befd-ec1d96fae29b" (UID: "d1ca0ea9-f1f1-48d7-befd-ec1d96fae29b"). InnerVolumeSpecName "kube-api-access-hs6wj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:44:26.386311 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:26.386231    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7460-140212187043568"]
Sep 09 15:44:26.386664 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:26.386651    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1ca0ea9-f1f1-48d7-befd-ec1d96fae29b" containerName="test-curl-7460-140212187043568"
Sep 09 15:44:26.386717 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:26.386666    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1ca0ea9-f1f1-48d7-befd-ec1d96fae29b" containerName="test-curl-7460-140212187043568"
Sep 09 15:44:26.386777 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:26.386724    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1ca0ea9-f1f1-48d7-befd-ec1d96fae29b" containerName="test-curl-7460-140212187043568"
Sep 09 15:44:26.389786 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:26.389770    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7460-140212187043568"
Sep 09 15:44:26.395925 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:26.395893    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs6wj\" (UniqueName: \"kubernetes.io/projected/d1ca0ea9-f1f1-48d7-befd-ec1d96fae29b-kube-api-access-hs6wj\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:44:26.397184 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:26.397161    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7460-140212187043568"]
Sep 09 15:44:26.496838 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:26.496800    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbl8h\" (UniqueName: \"kubernetes.io/projected/6b0250e6-5b85-4c7c-92db-c42d91f54317-kube-api-access-tbl8h\") pod \"test-curl-7460-140212187043568\" (UID: \"6b0250e6-5b85-4c7c-92db-c42d91f54317\") " pod="opendatahub/test-curl-7460-140212187043568"
Sep 09 15:44:26.598478 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:26.598434    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbl8h\" (UniqueName: \"kubernetes.io/projected/6b0250e6-5b85-4c7c-92db-c42d91f54317-kube-api-access-tbl8h\") pod \"test-curl-7460-140212187043568\" (UID: \"6b0250e6-5b85-4c7c-92db-c42d91f54317\") " pod="opendatahub/test-curl-7460-140212187043568"
Sep 09 15:44:26.607057 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:26.607025    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbl8h\" (UniqueName: \"kubernetes.io/projected/6b0250e6-5b85-4c7c-92db-c42d91f54317-kube-api-access-tbl8h\") pod \"test-curl-7460-140212187043568\" (UID: \"6b0250e6-5b85-4c7c-92db-c42d91f54317\") " pod="opendatahub/test-curl-7460-140212187043568"
Sep 09 15:44:26.701222 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:26.701184    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7460-140212187043568"
Sep 09 15:44:26.963392 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:26.963271    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1ca0ea9-f1f1-48d7-befd-ec1d96fae29b" path="/var/lib/kubelet/pods/d1ca0ea9-f1f1-48d7-befd-ec1d96fae29b/volumes"
Sep 09 15:44:26.972970 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:26.972944    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7460-140212187043568"]
Sep 09 15:44:26.975454 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:44:26.975432    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b0250e6_5b85_4c7c_92db_c42d91f54317.slice/crio-d1e2372558cc82e94cb578de7465287055535326a56454faa19900a3746621ef WatchSource:0}: Error finding container d1e2372558cc82e94cb578de7465287055535326a56454faa19900a3746621ef: Status 404 returned error can't find the container with id d1e2372558cc82e94cb578de7465287055535326a56454faa19900a3746621ef
Sep 09 15:44:27.067693 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:27.067656    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7460-140212187043568" event={"ID":"6b0250e6-5b85-4c7c-92db-c42d91f54317","Type":"ContainerStarted","Data":"d1e2372558cc82e94cb578de7465287055535326a56454faa19900a3746621ef"}
Sep 09 15:44:27.068862 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:27.068842    2553 scope.go:117] "RemoveContainer" containerID="13bd64cd179914b0e3aee253a4b8f4e83f6a732998f49f899b6c6ee6415e76b4"
Sep 09 15:44:27.068962 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:27.068873    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7460-140212187043568"
Sep 09 15:44:28.077074 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:28.077034    2553 generic.go:356] "Generic (PLEG): container finished" podID="6b0250e6-5b85-4c7c-92db-c42d91f54317" containerID="93e40d7a2f075fa65990dcacf4684e4a4c41f2b9689c33fd0e026a03e8122453" exitCode=0
Sep 09 15:44:28.077464 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:28.077106    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7460-140212187043568" event={"ID":"6b0250e6-5b85-4c7c-92db-c42d91f54317","Type":"ContainerDied","Data":"93e40d7a2f075fa65990dcacf4684e4a4c41f2b9689c33fd0e026a03e8122453"}
Sep 09 15:44:29.229319 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:29.229289    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7460-140212187043568"
Sep 09 15:44:29.250409 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:29.250372    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7460-140212187043568_6b0250e6-5b85-4c7c-92db-c42d91f54317/test-curl-7460-140212187043568/0.log"
Sep 09 15:44:29.274775 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:29.274726    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7460-140212187043568"]
Sep 09 15:44:29.278589 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:29.278564    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7460-140212187043568"]
Sep 09 15:44:29.328908 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:29.328877    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbl8h\" (UniqueName: \"kubernetes.io/projected/6b0250e6-5b85-4c7c-92db-c42d91f54317-kube-api-access-tbl8h\") pod \"6b0250e6-5b85-4c7c-92db-c42d91f54317\" (UID: \"6b0250e6-5b85-4c7c-92db-c42d91f54317\") "
Sep 09 15:44:29.330959 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:29.330926    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b0250e6-5b85-4c7c-92db-c42d91f54317-kube-api-access-tbl8h" (OuterVolumeSpecName: "kube-api-access-tbl8h") pod "6b0250e6-5b85-4c7c-92db-c42d91f54317" (UID: "6b0250e6-5b85-4c7c-92db-c42d91f54317"). InnerVolumeSpecName "kube-api-access-tbl8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:44:29.430448 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:29.430403    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbl8h\" (UniqueName: \"kubernetes.io/projected/6b0250e6-5b85-4c7c-92db-c42d91f54317-kube-api-access-tbl8h\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:44:30.089771 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:30.089743    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7460-140212187043568"
Sep 09 15:44:30.089950 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:30.089787    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1e2372558cc82e94cb578de7465287055535326a56454faa19900a3746621ef"
Sep 09 15:44:30.099660 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:44:30.099632    2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7460-140212187043568\" is forbidden: User \"system:node:ip-10-0-129-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-170.ec2.internal' and this object" podUID="6b0250e6-5b85-4c7c-92db-c42d91f54317" pod="opendatahub/test-curl-7460-140212187043568"
Sep 09 15:44:30.964045 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:30.964006    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b0250e6-5b85-4c7c-92db-c42d91f54317" path="/var/lib/kubelet/pods/6b0250e6-5b85-4c7c-92db-c42d91f54317/volumes"
Sep 09 15:44:34.957069 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:34.957036    2553 scope.go:117] "RemoveContainer" containerID="12b981f6562781df154b7148d41d393180d2538c08f5af66921367f4bcc31a43"
Sep 09 15:44:35.120822 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:35.120791    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-aeb0f8ca-bfb9bf8f7-cq8m8"]
Sep 09 15:44:35.121165 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:35.121153    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b0250e6-5b85-4c7c-92db-c42d91f54317" containerName="test-curl-7460-140212187043568"
Sep 09 15:44:35.121207 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:35.121167    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b0250e6-5b85-4c7c-92db-c42d91f54317" containerName="test-curl-7460-140212187043568"
Sep 09 15:44:35.121255 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:35.121245    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b0250e6-5b85-4c7c-92db-c42d91f54317" containerName="test-curl-7460-140212187043568"
Sep 09 15:44:35.125310 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:35.125293    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-aeb0f8ca-bfb9bf8f7-cq8m8"
Sep 09 15:44:35.127832 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:35.127810    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-aeb0f8ca\""
Sep 09 15:44:35.127960 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:35.127843    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-aeb0f8ca-dockercfg-cr8cz\""
Sep 09 15:44:35.135432 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:35.135409    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-aeb0f8ca-bfb9bf8f7-cq8m8"]
Sep 09 15:44:35.289098 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:35.288998    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee78420e-573b-4064-8797-8b86d24fdfc0-plugins-config-volume\") pod \"payload-processing-e2e-mt-aeb0f8ca-bfb9bf8f7-cq8m8\" (UID: \"ee78420e-573b-4064-8797-8b86d24fdfc0\") " pod="openshift-ingress/payload-processing-e2e-mt-aeb0f8ca-bfb9bf8f7-cq8m8"
Sep 09 15:44:35.289098 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:35.289039    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crz4p\" (UniqueName: \"kubernetes.io/projected/ee78420e-573b-4064-8797-8b86d24fdfc0-kube-api-access-crz4p\") pod \"payload-processing-e2e-mt-aeb0f8ca-bfb9bf8f7-cq8m8\" (UID: \"ee78420e-573b-4064-8797-8b86d24fdfc0\") " pod="openshift-ingress/payload-processing-e2e-mt-aeb0f8ca-bfb9bf8f7-cq8m8"
Sep 09 15:44:35.389907 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:35.389869    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee78420e-573b-4064-8797-8b86d24fdfc0-plugins-config-volume\") pod \"payload-processing-e2e-mt-aeb0f8ca-bfb9bf8f7-cq8m8\" (UID: \"ee78420e-573b-4064-8797-8b86d24fdfc0\") " pod="openshift-ingress/payload-processing-e2e-mt-aeb0f8ca-bfb9bf8f7-cq8m8"
Sep 09 15:44:35.389907 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:35.389907    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crz4p\" (UniqueName: \"kubernetes.io/projected/ee78420e-573b-4064-8797-8b86d24fdfc0-kube-api-access-crz4p\") pod \"payload-processing-e2e-mt-aeb0f8ca-bfb9bf8f7-cq8m8\" (UID: \"ee78420e-573b-4064-8797-8b86d24fdfc0\") " pod="openshift-ingress/payload-processing-e2e-mt-aeb0f8ca-bfb9bf8f7-cq8m8"
Sep 09 15:44:35.390555 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:35.390535    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee78420e-573b-4064-8797-8b86d24fdfc0-plugins-config-volume\") pod \"payload-processing-e2e-mt-aeb0f8ca-bfb9bf8f7-cq8m8\" (UID: \"ee78420e-573b-4064-8797-8b86d24fdfc0\") " pod="openshift-ingress/payload-processing-e2e-mt-aeb0f8ca-bfb9bf8f7-cq8m8"
Sep 09 15:44:35.398334 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:35.398307    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crz4p\" (UniqueName: \"kubernetes.io/projected/ee78420e-573b-4064-8797-8b86d24fdfc0-kube-api-access-crz4p\") pod \"payload-processing-e2e-mt-aeb0f8ca-bfb9bf8f7-cq8m8\" (UID: \"ee78420e-573b-4064-8797-8b86d24fdfc0\") " pod="openshift-ingress/payload-processing-e2e-mt-aeb0f8ca-bfb9bf8f7-cq8m8"
Sep 09 15:44:35.437139 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:35.437102    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-aeb0f8ca-bfb9bf8f7-cq8m8"
Sep 09 15:44:35.715795 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:35.715770    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-aeb0f8ca-bfb9bf8f7-cq8m8"]
Sep 09 15:44:35.717980 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:44:35.717951    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee78420e_573b_4064_8797_8b86d24fdfc0.slice/crio-5e5800611dc374626f000c8ab646b491e351c939890059ee9ab27c3a44c3191c WatchSource:0}: Error finding container 5e5800611dc374626f000c8ab646b491e351c939890059ee9ab27c3a44c3191c: Status 404 returned error can't find the container with id 5e5800611dc374626f000c8ab646b491e351c939890059ee9ab27c3a44c3191c
Sep 09 15:44:36.127149 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:36.127042    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-aeb0f8ca-bfb9bf8f7-cq8m8" event={"ID":"ee78420e-573b-4064-8797-8b86d24fdfc0","Type":"ContainerStarted","Data":"4e53c5f79974e6fe918cf706cbbcb130011cd60e7d12d086c044e5df14c2cc9d"}
Sep 09 15:44:36.127149 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:36.127106    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-aeb0f8ca-bfb9bf8f7-cq8m8"
Sep 09 15:44:36.127149 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:36.127134    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-aeb0f8ca-bfb9bf8f7-cq8m8" event={"ID":"ee78420e-573b-4064-8797-8b86d24fdfc0","Type":"ContainerStarted","Data":"5e5800611dc374626f000c8ab646b491e351c939890059ee9ab27c3a44c3191c"}
Sep 09 15:44:36.144959 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:36.144906    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-aeb0f8ca-bfb9bf8f7-cq8m8" podStartSLOduration=1.144884498 podStartE2EDuration="1.144884498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:44:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:44:36.143034348 +0000 UTC m=+1441.741928991" watchObservedRunningTime="2026-09-09 15:44:36.144884498 +0000 UTC m=+1441.743779199"
Sep 09 15:44:47.134887 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:47.134853    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-aeb0f8ca-bfb9bf8f7-cq8m8"
Sep 09 15:44:52.719744 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:52.719704    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57b579-svd29"]
Sep 09 15:44:52.727973 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:52.727948    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57b579-svd29"
Sep 09 15:44:52.731627 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:52.731536    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2zlsf\""
Sep 09 15:44:52.731627 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:52.731560    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 15:44:52.733561 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:52.733538    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57b579-svd29"]
Sep 09 15:44:52.760236 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:52.760196    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dabf7e2a-9b68-4e76-b9e9-be538e9dcde9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-57b579-svd29\" (UID: \"dabf7e2a-9b68-4e76-b9e9-be538e9dcde9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57b579-svd29"
Sep 09 15:44:52.861548 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:52.861513    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dabf7e2a-9b68-4e76-b9e9-be538e9dcde9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-57b579-svd29\" (UID: \"dabf7e2a-9b68-4e76-b9e9-be538e9dcde9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57b579-svd29"
Sep 09 15:44:52.862187 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:52.862166    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dabf7e2a-9b68-4e76-b9e9-be538e9dcde9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-57b579-svd29\" (UID: \"dabf7e2a-9b68-4e76-b9e9-be538e9dcde9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57b579-svd29"
Sep 09 15:44:53.039932 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:53.039834    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57b579-svd29"
Sep 09 15:44:53.317853 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:53.317825    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57b579-svd29"]
Sep 09 15:44:53.319708 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:44:53.319677    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddabf7e2a_9b68_4e76_b9e9_be538e9dcde9.slice/crio-03eb0349ee9f49e80eeba509f2b5950a43ce29cb1172558f17cf823aebbb3e77 WatchSource:0}: Error finding container 03eb0349ee9f49e80eeba509f2b5950a43ce29cb1172558f17cf823aebbb3e77: Status 404 returned error can't find the container with id 03eb0349ee9f49e80eeba509f2b5950a43ce29cb1172558f17cf823aebbb3e77
Sep 09 15:44:54.234379 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:54.233472    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57b579-svd29" event={"ID":"dabf7e2a-9b68-4e76-b9e9-be538e9dcde9","Type":"ContainerStarted","Data":"a74daae4ba0a553e0ad5dbc450b0b9718273bf07c2593ba09afe5f713a70fd27"}
Sep 09 15:44:54.234379 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:54.233540    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57b579-svd29" event={"ID":"dabf7e2a-9b68-4e76-b9e9-be538e9dcde9","Type":"ContainerStarted","Data":"03eb0349ee9f49e80eeba509f2b5950a43ce29cb1172558f17cf823aebbb3e77"}
Sep 09 15:44:54.249819 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:54.249773    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57b579-svd29" podStartSLOduration=2.249755732 podStartE2EDuration="2.249755732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:44:54.248569643 +0000 UTC m=+1459.847464303" watchObservedRunningTime="2026-09-09 15:44:54.249755732 +0000 UTC m=+1459.848650376"
Sep 09 15:44:55.242062 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:55.242030    2553 generic.go:356] "Generic (PLEG): container finished" podID="dabf7e2a-9b68-4e76-b9e9-be538e9dcde9" containerID="a74daae4ba0a553e0ad5dbc450b0b9718273bf07c2593ba09afe5f713a70fd27" exitCode=0
Sep 09 15:44:55.242429 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:55.242125    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57b579-svd29" event={"ID":"dabf7e2a-9b68-4e76-b9e9-be538e9dcde9","Type":"ContainerDied","Data":"a74daae4ba0a553e0ad5dbc450b0b9718273bf07c2593ba09afe5f713a70fd27"}
Sep 09 15:44:56.392026 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:56.391998    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57b579-svd29"
Sep 09 15:44:56.501880 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:56.501793    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dabf7e2a-9b68-4e76-b9e9-be538e9dcde9-maas-api-service-ca\") pod \"dabf7e2a-9b68-4e76-b9e9-be538e9dcde9\" (UID: \"dabf7e2a-9b68-4e76-b9e9-be538e9dcde9\") "
Sep 09 15:44:56.502143 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:56.502118    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dabf7e2a-9b68-4e76-b9e9-be538e9dcde9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dabf7e2a-9b68-4e76-b9e9-be538e9dcde9" (UID: "dabf7e2a-9b68-4e76-b9e9-be538e9dcde9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:44:56.603242 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:56.603202    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dabf7e2a-9b68-4e76-b9e9-be538e9dcde9-maas-api-service-ca\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:44:57.255013 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:57.254934    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57b579-svd29"
Sep 09 15:44:57.255013 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:57.254952    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57b579-svd29" event={"ID":"dabf7e2a-9b68-4e76-b9e9-be538e9dcde9","Type":"ContainerDied","Data":"03eb0349ee9f49e80eeba509f2b5950a43ce29cb1172558f17cf823aebbb3e77"}
Sep 09 15:44:57.255013 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:44:57.254995    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03eb0349ee9f49e80eeba509f2b5950a43ce29cb1172558f17cf823aebbb3e77"
Sep 09 15:45:00.142488 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:00.142447    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aeb0f8ca-29816145-7s6dx"]
Sep 09 15:45:00.143105 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:00.143083    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dabf7e2a-9b68-4e76-b9e9-be538e9dcde9" containerName="revoke-keys"
Sep 09 15:45:00.143186 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:00.143107    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="dabf7e2a-9b68-4e76-b9e9-be538e9dcde9" containerName="revoke-keys"
Sep 09 15:45:00.143255 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:00.143241    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="dabf7e2a-9b68-4e76-b9e9-be538e9dcde9" containerName="revoke-keys"
Sep 09 15:45:00.154892 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:00.154821    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1e15c8-29816145-kzt6s"]
Sep 09 15:45:00.155050 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:00.154916    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aeb0f8ca-29816145-7s6dx"
Sep 09 15:45:00.158918 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:00.158890    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2zlsf\""
Sep 09 15:45:00.159069 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:00.159006    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-267962-29816145-bg6vw"]
Sep 09 15:45:00.159263 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:00.159241    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1e15c8-29816145-kzt6s"
Sep 09 15:45:00.164452 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:00.164428    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aeb0f8ca-29816145-7s6dx"]
Sep 09 15:45:00.164797 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:00.164779    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-267962-29816145-bg6vw"
Sep 09 15:45:00.171360 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:00.171251    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1e15c8-29816145-kzt6s"]
Sep 09 15:45:00.173474 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:00.173451    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aeb0f8ca-29816145-7s6dx"
Sep 09 15:45:00.174356 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:00.174337    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816145-7nbjw"]
Sep 09 15:45:00.178511 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:00.178483    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-081ed3-29816145-qk6nx"]
Sep 09 15:45:00.178791 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:00.178756    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816145-7nbjw"
Sep 09 15:45:00.182928 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:00.182904    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1e15c8-29816145-kzt6s"
Sep 09 15:45:00.184015 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:00.183907    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-081ed3-29816145-qk6nx"
Sep 09 15:45:00.192065 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:00.192027    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816145-7nbjw"]
Sep 09 15:45:00.193797 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:00.193772    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-267962-29816145-bg6vw"]
Sep 09 15:45:00.210128 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:00.209610    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-081ed3-29816145-qk6nx"]
Sep 09 15:45:00.240834 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:00.240800    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-267962-29816145-bg6vw"
Sep 09 15:45:00.257582 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:00.257557    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816145-7nbjw"
Sep 09 15:45:00.288335 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:00.288290    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-081ed3-29816145-qk6nx"
Sep 09 15:45:00.476783 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:00.476754    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aeb0f8ca-29816145-7s6dx"]
Sep 09 15:45:00.479079 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:45:00.479048    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod064b7c73_ea65_4dec_95b3_607237686388.slice/crio-13b81f8cb6ef2524d638ff17651ce0f0cec09047e7ad874689f5625cf8cea53f WatchSource:0}: Error finding container 13b81f8cb6ef2524d638ff17651ce0f0cec09047e7ad874689f5625cf8cea53f: Status 404 returned error can't find the container with id 13b81f8cb6ef2524d638ff17651ce0f0cec09047e7ad874689f5625cf8cea53f
Sep 09 15:45:00.497947 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:00.497920    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1e15c8-29816145-kzt6s"]
Sep 09 15:45:00.499481 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:45:00.499457    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99226359_e48c_455a_9c80_4e8c21dae0a4.slice/crio-774a7f7a58c96138a67bee2523ffc56027774f113054d3f2ed8742e93b8b2087 WatchSource:0}: Error finding container 774a7f7a58c96138a67bee2523ffc56027774f113054d3f2ed8742e93b8b2087: Status 404 returned error can't find the container with id 774a7f7a58c96138a67bee2523ffc56027774f113054d3f2ed8742e93b8b2087
Sep 09 15:45:00.553062 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:00.553038    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-267962-29816145-bg6vw"]
Sep 09 15:45:00.554575 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:45:00.554545    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37439468_c16a_49ad_af2d_724f3fc4c9ff.slice/crio-50dae5de73756ce82ea2576da4484710bb3e8ce39feed595e78d8b08149099d6 WatchSource:0}: Error finding container 50dae5de73756ce82ea2576da4484710bb3e8ce39feed595e78d8b08149099d6: Status 404 returned error can't find the container with id 50dae5de73756ce82ea2576da4484710bb3e8ce39feed595e78d8b08149099d6
Sep 09 15:45:00.575698 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:00.575673    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816145-7nbjw"]
Sep 09 15:45:00.577953 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:45:00.577932    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4de61e9c_6f7b_4aec_8793_b679da741a84.slice/crio-be01fe70e59172a102bea266cd79cda3945ffbf3eb5782a48f2805d7ec78e380 WatchSource:0}: Error finding container be01fe70e59172a102bea266cd79cda3945ffbf3eb5782a48f2805d7ec78e380: Status 404 returned error can't find the container with id be01fe70e59172a102bea266cd79cda3945ffbf3eb5782a48f2805d7ec78e380
Sep 09 15:45:00.580418 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:00.580397    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-081ed3-29816145-qk6nx"]
Sep 09 15:45:00.585742 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:45:00.585700    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38de0606_7eb9_4429_8c05_2d9a8fa23465.slice/crio-5c9a7e51f6453abd1898dbb4f4d77c6d5554e025b026bfbdcb1d7ceb268e8dab WatchSource:0}: Error finding container 5c9a7e51f6453abd1898dbb4f4d77c6d5554e025b026bfbdcb1d7ceb268e8dab: Status 404 returned error can't find the container with id 5c9a7e51f6453abd1898dbb4f4d77c6d5554e025b026bfbdcb1d7ceb268e8dab
Sep 09 15:45:01.298588 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:01.298544    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816145-7nbjw" event={"ID":"4de61e9c-6f7b-4aec-8793-b679da741a84","Type":"ContainerStarted","Data":"a9adb1158a365b0191e3085180989d33fbe6b074781294c42089cdbddd0cc88f"}
Sep 09 15:45:01.299048 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:01.298607    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816145-7nbjw" event={"ID":"4de61e9c-6f7b-4aec-8793-b679da741a84","Type":"ContainerStarted","Data":"be01fe70e59172a102bea266cd79cda3945ffbf3eb5782a48f2805d7ec78e380"}
Sep 09 15:45:01.300433 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:01.300404    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aeb0f8ca-29816145-7s6dx" event={"ID":"064b7c73-ea65-4dec-95b3-607237686388","Type":"ContainerStarted","Data":"b8c06d300d135a4337a276584e4d9f178733aac32ccb748a260cce70cdf791f0"}
Sep 09 15:45:01.300567 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:01.300458    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aeb0f8ca-29816145-7s6dx" event={"ID":"064b7c73-ea65-4dec-95b3-607237686388","Type":"ContainerStarted","Data":"13b81f8cb6ef2524d638ff17651ce0f0cec09047e7ad874689f5625cf8cea53f"}
Sep 09 15:45:01.301901 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:01.301873    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-267962-29816145-bg6vw" event={"ID":"37439468-c16a-49ad-af2d-724f3fc4c9ff","Type":"ContainerStarted","Data":"51c902e1bfe7b0c55d9ddd47992b8667e2568b6995a8e3b1156e85f943a358dd"}
Sep 09 15:45:01.302031 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:01.301930    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-267962-29816145-bg6vw" event={"ID":"37439468-c16a-49ad-af2d-724f3fc4c9ff","Type":"ContainerStarted","Data":"50dae5de73756ce82ea2576da4484710bb3e8ce39feed595e78d8b08149099d6"}
Sep 09 15:45:01.303460 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:01.303439    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1e15c8-29816145-kzt6s" event={"ID":"99226359-e48c-455a-9c80-4e8c21dae0a4","Type":"ContainerStarted","Data":"be089da702bd8dfdfa25390aa0cd05a00ae9306afd3916fb83794f997f0c1208"}
Sep 09 15:45:01.303544 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:01.303479    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1e15c8-29816145-kzt6s" event={"ID":"99226359-e48c-455a-9c80-4e8c21dae0a4","Type":"ContainerStarted","Data":"774a7f7a58c96138a67bee2523ffc56027774f113054d3f2ed8742e93b8b2087"}
Sep 09 15:45:01.304895 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:01.304873    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-081ed3-29816145-qk6nx" event={"ID":"38de0606-7eb9-4429-8c05-2d9a8fa23465","Type":"ContainerStarted","Data":"c170b7a062ef4cf28e736f0a75135719ecaada8258ed67c315be586aafa7866d"}
Sep 09 15:45:01.304978 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:01.304922    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-081ed3-29816145-qk6nx" event={"ID":"38de0606-7eb9-4429-8c05-2d9a8fa23465","Type":"ContainerStarted","Data":"5c9a7e51f6453abd1898dbb4f4d77c6d5554e025b026bfbdcb1d7ceb268e8dab"}
Sep 09 15:45:01.314047 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:01.314005    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816145-7nbjw" podStartSLOduration=1.313992 podStartE2EDuration="1.313992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:45:01.312521936 +0000 UTC m=+1466.911416580" watchObservedRunningTime="2026-09-09 15:45:01.313992 +0000 UTC m=+1466.912886671"
Sep 09 15:45:01.327481 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:01.327429    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-267962-29816145-bg6vw" podStartSLOduration=1.327416264 podStartE2EDuration="1.327416264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:45:01.325539491 +0000 UTC m=+1466.924434136" watchObservedRunningTime="2026-09-09 15:45:01.327416264 +0000 UTC m=+1466.926310908"
Sep 09 15:45:01.359179 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:01.359115    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1e15c8-29816145-kzt6s" podStartSLOduration=1.359098657 podStartE2EDuration="1.359098657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:45:01.341253163 +0000 UTC m=+1466.940147809" watchObservedRunningTime="2026-09-09 15:45:01.359098657 +0000 UTC m=+1466.957993301"
Sep 09 15:45:01.372095 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:01.372045    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aeb0f8ca-29816145-7s6dx" podStartSLOduration=1.372027689 podStartE2EDuration="1.372027689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:45:01.357717185 +0000 UTC m=+1466.956611829" watchObservedRunningTime="2026-09-09 15:45:01.372027689 +0000 UTC m=+1466.970922352"
Sep 09 15:45:01.373407 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:01.373371    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-081ed3-29816145-qk6nx" podStartSLOduration=1.373359135 podStartE2EDuration="1.373359135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:45:01.371861676 +0000 UTC m=+1466.970756319" watchObservedRunningTime="2026-09-09 15:45:01.373359135 +0000 UTC m=+1466.972253819"
Sep 09 15:45:02.312613 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:02.312508    2553 generic.go:356] "Generic (PLEG): container finished" podID="064b7c73-ea65-4dec-95b3-607237686388" containerID="b8c06d300d135a4337a276584e4d9f178733aac32ccb748a260cce70cdf791f0" exitCode=0
Sep 09 15:45:02.312613 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:02.312589    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aeb0f8ca-29816145-7s6dx" event={"ID":"064b7c73-ea65-4dec-95b3-607237686388","Type":"ContainerDied","Data":"b8c06d300d135a4337a276584e4d9f178733aac32ccb748a260cce70cdf791f0"}
Sep 09 15:45:02.314256 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:02.314227    2553 generic.go:356] "Generic (PLEG): container finished" podID="37439468-c16a-49ad-af2d-724f3fc4c9ff" containerID="51c902e1bfe7b0c55d9ddd47992b8667e2568b6995a8e3b1156e85f943a358dd" exitCode=0
Sep 09 15:45:02.314352 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:02.314302    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-267962-29816145-bg6vw" event={"ID":"37439468-c16a-49ad-af2d-724f3fc4c9ff","Type":"ContainerDied","Data":"51c902e1bfe7b0c55d9ddd47992b8667e2568b6995a8e3b1156e85f943a358dd"}
Sep 09 15:45:02.315796 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:02.315776    2553 generic.go:356] "Generic (PLEG): container finished" podID="99226359-e48c-455a-9c80-4e8c21dae0a4" containerID="be089da702bd8dfdfa25390aa0cd05a00ae9306afd3916fb83794f997f0c1208" exitCode=0
Sep 09 15:45:02.315881 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:02.315803    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1e15c8-29816145-kzt6s" event={"ID":"99226359-e48c-455a-9c80-4e8c21dae0a4","Type":"ContainerDied","Data":"be089da702bd8dfdfa25390aa0cd05a00ae9306afd3916fb83794f997f0c1208"}
Sep 09 15:45:02.317466 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:02.317449    2553 generic.go:356] "Generic (PLEG): container finished" podID="38de0606-7eb9-4429-8c05-2d9a8fa23465" containerID="c170b7a062ef4cf28e736f0a75135719ecaada8258ed67c315be586aafa7866d" exitCode=0
Sep 09 15:45:02.317554 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:02.317521    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-081ed3-29816145-qk6nx" event={"ID":"38de0606-7eb9-4429-8c05-2d9a8fa23465","Type":"ContainerDied","Data":"c170b7a062ef4cf28e736f0a75135719ecaada8258ed67c315be586aafa7866d"}
Sep 09 15:45:02.319038 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:02.319021    2553 generic.go:356] "Generic (PLEG): container finished" podID="4de61e9c-6f7b-4aec-8793-b679da741a84" containerID="a9adb1158a365b0191e3085180989d33fbe6b074781294c42089cdbddd0cc88f" exitCode=0
Sep 09 15:45:02.319100 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:02.319052    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816145-7nbjw" event={"ID":"4de61e9c-6f7b-4aec-8793-b679da741a84","Type":"ContainerDied","Data":"a9adb1158a365b0191e3085180989d33fbe6b074781294c42089cdbddd0cc88f"}
Sep 09 15:45:03.487935 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.487909    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aeb0f8ca-29816145-7s6dx"
Sep 09 15:45:03.526354 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.526179    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816145-7nbjw"
Sep 09 15:45:03.579801 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.579725    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-267962-29816145-bg6vw"
Sep 09 15:45:03.583409 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.583384    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1e15c8-29816145-kzt6s"
Sep 09 15:45:03.586857 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.586830    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-081ed3-29816145-qk6nx"
Sep 09 15:45:03.676000 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.675965    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg"]
Sep 09 15:45:03.676378 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.676365    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99226359-e48c-455a-9c80-4e8c21dae0a4" containerName="cleanup"
Sep 09 15:45:03.676425 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.676379    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="99226359-e48c-455a-9c80-4e8c21dae0a4" containerName="cleanup"
Sep 09 15:45:03.676425 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.676394    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4de61e9c-6f7b-4aec-8793-b679da741a84" containerName="cleanup"
Sep 09 15:45:03.676425 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.676400    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="4de61e9c-6f7b-4aec-8793-b679da741a84" containerName="cleanup"
Sep 09 15:45:03.676425 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.676408    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="064b7c73-ea65-4dec-95b3-607237686388" containerName="cleanup"
Sep 09 15:45:03.676425 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.676413    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="064b7c73-ea65-4dec-95b3-607237686388" containerName="cleanup"
Sep 09 15:45:03.676425 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.676419    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37439468-c16a-49ad-af2d-724f3fc4c9ff" containerName="cleanup"
Sep 09 15:45:03.676425 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.676424    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="37439468-c16a-49ad-af2d-724f3fc4c9ff" containerName="cleanup"
Sep 09 15:45:03.676626 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.676431    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38de0606-7eb9-4429-8c05-2d9a8fa23465" containerName="cleanup"
Sep 09 15:45:03.676626 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.676437    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="38de0606-7eb9-4429-8c05-2d9a8fa23465" containerName="cleanup"
Sep 09 15:45:03.676626 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.676504    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="4de61e9c-6f7b-4aec-8793-b679da741a84" containerName="cleanup"
Sep 09 15:45:03.676626 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.676512    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="38de0606-7eb9-4429-8c05-2d9a8fa23465" containerName="cleanup"
Sep 09 15:45:03.676626 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.676518    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="37439468-c16a-49ad-af2d-724f3fc4c9ff" containerName="cleanup"
Sep 09 15:45:03.676626 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.676524    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="064b7c73-ea65-4dec-95b3-607237686388" containerName="cleanup"
Sep 09 15:45:03.676626 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.676529    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="99226359-e48c-455a-9c80-4e8c21dae0a4" containerName="cleanup"
Sep 09 15:45:03.679571 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.679553    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1e15c8/distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg"
Sep 09 15:45:03.682012 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.681990    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1e15c8\"/\"distinct-2-w4-1e15c8-kserve-self-signed-certs\""
Sep 09 15:45:03.687002 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.686979    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg"]
Sep 09 15:45:03.775285 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.775249    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd4ed05e-a093-4371-9e6b-6983225f91b5-model-cache\") pod \"distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg\" (UID: \"cd4ed05e-a093-4371-9e6b-6983225f91b5\") " pod="e2e-models-e2e-worker-w4-1e15c8/distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg"
Sep 09 15:45:03.775427 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.775299    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd4ed05e-a093-4371-9e6b-6983225f91b5-tmp-dir\") pod \"distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg\" (UID: \"cd4ed05e-a093-4371-9e6b-6983225f91b5\") " pod="e2e-models-e2e-worker-w4-1e15c8/distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg"
Sep 09 15:45:03.775484 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.775436    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92v7b\" (UniqueName: \"kubernetes.io/projected/cd4ed05e-a093-4371-9e6b-6983225f91b5-kube-api-access-92v7b\") pod \"distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg\" (UID: \"cd4ed05e-a093-4371-9e6b-6983225f91b5\") " pod="e2e-models-e2e-worker-w4-1e15c8/distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg"
Sep 09 15:45:03.775484 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.775476    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd4ed05e-a093-4371-9e6b-6983225f91b5-home\") pod \"distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg\" (UID: \"cd4ed05e-a093-4371-9e6b-6983225f91b5\") " pod="e2e-models-e2e-worker-w4-1e15c8/distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg"
Sep 09 15:45:03.775555 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.775544    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4ed05e-a093-4371-9e6b-6983225f91b5-tls-certs\") pod \"distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg\" (UID: \"cd4ed05e-a093-4371-9e6b-6983225f91b5\") " pod="e2e-models-e2e-worker-w4-1e15c8/distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg"
Sep 09 15:45:03.775606 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.775592    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd4ed05e-a093-4371-9e6b-6983225f91b5-dshm\") pod \"distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg\" (UID: \"cd4ed05e-a093-4371-9e6b-6983225f91b5\") " pod="e2e-models-e2e-worker-w4-1e15c8/distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg"
Sep 09 15:45:03.876253 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.876159    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd4ed05e-a093-4371-9e6b-6983225f91b5-model-cache\") pod \"distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg\" (UID: \"cd4ed05e-a093-4371-9e6b-6983225f91b5\") " pod="e2e-models-e2e-worker-w4-1e15c8/distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg"
Sep 09 15:45:03.876253 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.876205    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd4ed05e-a093-4371-9e6b-6983225f91b5-tmp-dir\") pod \"distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg\" (UID: \"cd4ed05e-a093-4371-9e6b-6983225f91b5\") " pod="e2e-models-e2e-worker-w4-1e15c8/distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg"
Sep 09 15:45:03.876479 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.876273    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92v7b\" (UniqueName: \"kubernetes.io/projected/cd4ed05e-a093-4371-9e6b-6983225f91b5-kube-api-access-92v7b\") pod \"distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg\" (UID: \"cd4ed05e-a093-4371-9e6b-6983225f91b5\") " pod="e2e-models-e2e-worker-w4-1e15c8/distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg"
Sep 09 15:45:03.876479 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.876303    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd4ed05e-a093-4371-9e6b-6983225f91b5-home\") pod \"distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg\" (UID: \"cd4ed05e-a093-4371-9e6b-6983225f91b5\") " pod="e2e-models-e2e-worker-w4-1e15c8/distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg"
Sep 09 15:45:03.876479 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.876348    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4ed05e-a093-4371-9e6b-6983225f91b5-tls-certs\") pod \"distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg\" (UID: \"cd4ed05e-a093-4371-9e6b-6983225f91b5\") " pod="e2e-models-e2e-worker-w4-1e15c8/distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg"
Sep 09 15:45:03.876479 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.876384    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd4ed05e-a093-4371-9e6b-6983225f91b5-dshm\") pod \"distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg\" (UID: \"cd4ed05e-a093-4371-9e6b-6983225f91b5\") " pod="e2e-models-e2e-worker-w4-1e15c8/distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg"
Sep 09 15:45:03.876679 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.876599    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd4ed05e-a093-4371-9e6b-6983225f91b5-model-cache\") pod \"distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg\" (UID: \"cd4ed05e-a093-4371-9e6b-6983225f91b5\") " pod="e2e-models-e2e-worker-w4-1e15c8/distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg"
Sep 09 15:45:03.876679 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.876652    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd4ed05e-a093-4371-9e6b-6983225f91b5-tmp-dir\") pod \"distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg\" (UID: \"cd4ed05e-a093-4371-9e6b-6983225f91b5\") " pod="e2e-models-e2e-worker-w4-1e15c8/distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg"
Sep 09 15:45:03.876679 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.876668    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd4ed05e-a093-4371-9e6b-6983225f91b5-home\") pod \"distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg\" (UID: \"cd4ed05e-a093-4371-9e6b-6983225f91b5\") " pod="e2e-models-e2e-worker-w4-1e15c8/distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg"
Sep 09 15:45:03.878639 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.878616    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd4ed05e-a093-4371-9e6b-6983225f91b5-dshm\") pod \"distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg\" (UID: \"cd4ed05e-a093-4371-9e6b-6983225f91b5\") " pod="e2e-models-e2e-worker-w4-1e15c8/distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg"
Sep 09 15:45:03.878838 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.878821    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4ed05e-a093-4371-9e6b-6983225f91b5-tls-certs\") pod \"distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg\" (UID: \"cd4ed05e-a093-4371-9e6b-6983225f91b5\") " pod="e2e-models-e2e-worker-w4-1e15c8/distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg"
Sep 09 15:45:03.884421 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.884403    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92v7b\" (UniqueName: \"kubernetes.io/projected/cd4ed05e-a093-4371-9e6b-6983225f91b5-kube-api-access-92v7b\") pod \"distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg\" (UID: \"cd4ed05e-a093-4371-9e6b-6983225f91b5\") " pod="e2e-models-e2e-worker-w4-1e15c8/distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg"
Sep 09 15:45:03.989833 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:03.989792    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1e15c8/distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg"
Sep 09 15:45:04.296841 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:04.296808    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg"]
Sep 09 15:45:04.303056 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:45:04.303028    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd4ed05e_a093_4371_9e6b_6983225f91b5.slice/crio-c6ff0a5349eebf4b733bdfedfd5d9ef05f1d89787de306bbe6b5ab753985d197 WatchSource:0}: Error finding container c6ff0a5349eebf4b733bdfedfd5d9ef05f1d89787de306bbe6b5ab753985d197: Status 404 returned error can't find the container with id c6ff0a5349eebf4b733bdfedfd5d9ef05f1d89787de306bbe6b5ab753985d197
Sep 09 15:45:04.333726 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:04.333704    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aeb0f8ca-29816145-7s6dx"
Sep 09 15:45:04.333846 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:04.333704    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aeb0f8ca-29816145-7s6dx" event={"ID":"064b7c73-ea65-4dec-95b3-607237686388","Type":"ContainerDied","Data":"13b81f8cb6ef2524d638ff17651ce0f0cec09047e7ad874689f5625cf8cea53f"}
Sep 09 15:45:04.333846 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:04.333786    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13b81f8cb6ef2524d638ff17651ce0f0cec09047e7ad874689f5625cf8cea53f"
Sep 09 15:45:04.335163 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:04.335135    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-267962-29816145-bg6vw"
Sep 09 15:45:04.335283 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:04.335177    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-267962-29816145-bg6vw" event={"ID":"37439468-c16a-49ad-af2d-724f3fc4c9ff","Type":"ContainerDied","Data":"50dae5de73756ce82ea2576da4484710bb3e8ce39feed595e78d8b08149099d6"}
Sep 09 15:45:04.335283 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:04.335216    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50dae5de73756ce82ea2576da4484710bb3e8ce39feed595e78d8b08149099d6"
Sep 09 15:45:04.336566 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:04.336531    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1e15c8/distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg" event={"ID":"cd4ed05e-a093-4371-9e6b-6983225f91b5","Type":"ContainerStarted","Data":"c6ff0a5349eebf4b733bdfedfd5d9ef05f1d89787de306bbe6b5ab753985d197"}
Sep 09 15:45:04.338644 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:04.338102    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1e15c8-29816145-kzt6s"
Sep 09 15:45:04.338644 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:04.338114    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1e15c8-29816145-kzt6s" event={"ID":"99226359-e48c-455a-9c80-4e8c21dae0a4","Type":"ContainerDied","Data":"774a7f7a58c96138a67bee2523ffc56027774f113054d3f2ed8742e93b8b2087"}
Sep 09 15:45:04.338644 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:04.338160    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="774a7f7a58c96138a67bee2523ffc56027774f113054d3f2ed8742e93b8b2087"
Sep 09 15:45:04.340854 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:04.340413    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-081ed3-29816145-qk6nx"
Sep 09 15:45:04.340854 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:04.340502    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-081ed3-29816145-qk6nx" event={"ID":"38de0606-7eb9-4429-8c05-2d9a8fa23465","Type":"ContainerDied","Data":"5c9a7e51f6453abd1898dbb4f4d77c6d5554e025b026bfbdcb1d7ceb268e8dab"}
Sep 09 15:45:04.340854 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:04.340547    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c9a7e51f6453abd1898dbb4f4d77c6d5554e025b026bfbdcb1d7ceb268e8dab"
Sep 09 15:45:04.342303 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:04.342279    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816145-7nbjw" event={"ID":"4de61e9c-6f7b-4aec-8793-b679da741a84","Type":"ContainerDied","Data":"be01fe70e59172a102bea266cd79cda3945ffbf3eb5782a48f2805d7ec78e380"}
Sep 09 15:45:04.342415 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:04.342325    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be01fe70e59172a102bea266cd79cda3945ffbf3eb5782a48f2805d7ec78e380"
Sep 09 15:45:04.342415 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:04.342373    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816145-7nbjw"
Sep 09 15:45:04.512394 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:04.512363    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-57b579-7994b7cc87-fgscr"]
Sep 09 15:45:04.512832 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:04.512602    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-57b579-7994b7cc87-fgscr" podUID="9d33f110-3b40-4098-8075-2a42dba9c751" containerName="payload-processing" containerID="cri-o://4d74934ef89406a1c0b42f768da7324874405e0274ad76bf6855898c009000b6" gracePeriod=30
Sep 09 15:45:04.521983 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:04.521955    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-57b579-58954c5b5d-dvsnc"]
Sep 09 15:45:04.522466 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:04.522395    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-57b579-58954c5b5d-dvsnc" podUID="015c0cd8-414e-4d52-aeee-c64c1588b315" containerName="payload-pre-processing" containerID="cri-o://25de6eb9b21dde21f7a1f94cb9adaf868d5d5805edbbb052eee0e955e6503f00" gracePeriod=30
Sep 09 15:45:04.691500 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:04.691450    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57b579-svd29"]
Sep 09 15:45:04.693973 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:04.693947    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57b579-svd29"]
Sep 09 15:45:04.793342 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:04.793304    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-57b579-7994b7cc87-fgscr"
Sep 09 15:45:04.816298 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:04.816268    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-57b579-58954c5b5d-dvsnc"
Sep 09 15:45:04.887420 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:04.887386    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d33f110-3b40-4098-8075-2a42dba9c751-plugins-config-volume\") pod \"9d33f110-3b40-4098-8075-2a42dba9c751\" (UID: \"9d33f110-3b40-4098-8075-2a42dba9c751\") "
Sep 09 15:45:04.887595 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:04.887428    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nh4kt\" (UniqueName: \"kubernetes.io/projected/9d33f110-3b40-4098-8075-2a42dba9c751-kube-api-access-nh4kt\") pod \"9d33f110-3b40-4098-8075-2a42dba9c751\" (UID: \"9d33f110-3b40-4098-8075-2a42dba9c751\") "
Sep 09 15:45:04.887639 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:04.887585    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lm8gh\" (UniqueName: \"kubernetes.io/projected/015c0cd8-414e-4d52-aeee-c64c1588b315-kube-api-access-lm8gh\") pod \"015c0cd8-414e-4d52-aeee-c64c1588b315\" (UID: \"015c0cd8-414e-4d52-aeee-c64c1588b315\") "
Sep 09 15:45:04.887722 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:04.887707    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/015c0cd8-414e-4d52-aeee-c64c1588b315-plugins-config-volume\") pod \"015c0cd8-414e-4d52-aeee-c64c1588b315\" (UID: \"015c0cd8-414e-4d52-aeee-c64c1588b315\") "
Sep 09 15:45:04.887901 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:04.887870    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d33f110-3b40-4098-8075-2a42dba9c751-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9d33f110-3b40-4098-8075-2a42dba9c751" (UID: "9d33f110-3b40-4098-8075-2a42dba9c751"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:45:04.888078 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:04.888053    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/015c0cd8-414e-4d52-aeee-c64c1588b315-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "015c0cd8-414e-4d52-aeee-c64c1588b315" (UID: "015c0cd8-414e-4d52-aeee-c64c1588b315"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:45:04.888204 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:04.888186    2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/015c0cd8-414e-4d52-aeee-c64c1588b315-plugins-config-volume\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:45:04.888251 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:04.888211    2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d33f110-3b40-4098-8075-2a42dba9c751-plugins-config-volume\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:45:04.889667 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:04.889644    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/015c0cd8-414e-4d52-aeee-c64c1588b315-kube-api-access-lm8gh" (OuterVolumeSpecName: "kube-api-access-lm8gh") pod "015c0cd8-414e-4d52-aeee-c64c1588b315" (UID: "015c0cd8-414e-4d52-aeee-c64c1588b315"). InnerVolumeSpecName "kube-api-access-lm8gh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:45:04.889753 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:04.889665    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d33f110-3b40-4098-8075-2a42dba9c751-kube-api-access-nh4kt" (OuterVolumeSpecName: "kube-api-access-nh4kt") pod "9d33f110-3b40-4098-8075-2a42dba9c751" (UID: "9d33f110-3b40-4098-8075-2a42dba9c751"). InnerVolumeSpecName "kube-api-access-nh4kt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:45:04.966401 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:04.966365    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dabf7e2a-9b68-4e76-b9e9-be538e9dcde9" path="/var/lib/kubelet/pods/dabf7e2a-9b68-4e76-b9e9-be538e9dcde9/volumes"
Sep 09 15:45:04.988924 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:04.988897    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh4kt\" (UniqueName: \"kubernetes.io/projected/9d33f110-3b40-4098-8075-2a42dba9c751-kube-api-access-nh4kt\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:45:04.988924 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:04.988924    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm8gh\" (UniqueName: \"kubernetes.io/projected/015c0cd8-414e-4d52-aeee-c64c1588b315-kube-api-access-lm8gh\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:45:05.349816 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:05.349778    2553 generic.go:356] "Generic (PLEG): container finished" podID="9d33f110-3b40-4098-8075-2a42dba9c751" containerID="4d74934ef89406a1c0b42f768da7324874405e0274ad76bf6855898c009000b6" exitCode=0
Sep 09 15:45:05.349997 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:05.349829    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-57b579-7994b7cc87-fgscr"
Sep 09 15:45:05.349997 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:05.349841    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-57b579-7994b7cc87-fgscr" event={"ID":"9d33f110-3b40-4098-8075-2a42dba9c751","Type":"ContainerDied","Data":"4d74934ef89406a1c0b42f768da7324874405e0274ad76bf6855898c009000b6"}
Sep 09 15:45:05.349997 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:05.349917    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-57b579-7994b7cc87-fgscr" event={"ID":"9d33f110-3b40-4098-8075-2a42dba9c751","Type":"ContainerDied","Data":"506d56f436354d916a6bb5463b0919d1fb5051e47d0644a154a24aa1fa12f4ef"}
Sep 09 15:45:05.349997 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:05.349950    2553 scope.go:117] "RemoveContainer" containerID="4d74934ef89406a1c0b42f768da7324874405e0274ad76bf6855898c009000b6"
Sep 09 15:45:05.352111 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:05.352084    2553 generic.go:356] "Generic (PLEG): container finished" podID="015c0cd8-414e-4d52-aeee-c64c1588b315" containerID="25de6eb9b21dde21f7a1f94cb9adaf868d5d5805edbbb052eee0e955e6503f00" exitCode=0
Sep 09 15:45:05.352317 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:05.352295    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-57b579-58954c5b5d-dvsnc"
Sep 09 15:45:05.352461 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:05.352293    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-57b579-58954c5b5d-dvsnc" event={"ID":"015c0cd8-414e-4d52-aeee-c64c1588b315","Type":"ContainerDied","Data":"25de6eb9b21dde21f7a1f94cb9adaf868d5d5805edbbb052eee0e955e6503f00"}
Sep 09 15:45:05.352529 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:05.352498    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-57b579-58954c5b5d-dvsnc" event={"ID":"015c0cd8-414e-4d52-aeee-c64c1588b315","Type":"ContainerDied","Data":"70bde9b2be9a03d8296cbaa8699ab021127fe7f9b68838d300623047905cfebc"}
Sep 09 15:45:05.354842 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:05.354508    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1e15c8/distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg" event={"ID":"cd4ed05e-a093-4371-9e6b-6983225f91b5","Type":"ContainerStarted","Data":"7f11e2aecaafa78df1f0787eee91fe5a6a9ca3630eb75b0ba47198ee2b208231"}
Sep 09 15:45:05.360625 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:05.360463    2553 scope.go:117] "RemoveContainer" containerID="4d74934ef89406a1c0b42f768da7324874405e0274ad76bf6855898c009000b6"
Sep 09 15:45:05.360835 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:45:05.360812    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d74934ef89406a1c0b42f768da7324874405e0274ad76bf6855898c009000b6\": container with ID starting with 4d74934ef89406a1c0b42f768da7324874405e0274ad76bf6855898c009000b6 not found: ID does not exist" containerID="4d74934ef89406a1c0b42f768da7324874405e0274ad76bf6855898c009000b6"
Sep 09 15:45:05.360932 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:05.360851    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d74934ef89406a1c0b42f768da7324874405e0274ad76bf6855898c009000b6"} err="failed to get container status \"4d74934ef89406a1c0b42f768da7324874405e0274ad76bf6855898c009000b6\": rpc error: code = NotFound desc = could not find container \"4d74934ef89406a1c0b42f768da7324874405e0274ad76bf6855898c009000b6\": container with ID starting with 4d74934ef89406a1c0b42f768da7324874405e0274ad76bf6855898c009000b6 not found: ID does not exist"
Sep 09 15:45:05.360932 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:05.360875    2553 scope.go:117] "RemoveContainer" containerID="25de6eb9b21dde21f7a1f94cb9adaf868d5d5805edbbb052eee0e955e6503f00"
Sep 09 15:45:05.369499 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:05.369480    2553 scope.go:117] "RemoveContainer" containerID="25de6eb9b21dde21f7a1f94cb9adaf868d5d5805edbbb052eee0e955e6503f00"
Sep 09 15:45:05.369844 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:45:05.369824    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25de6eb9b21dde21f7a1f94cb9adaf868d5d5805edbbb052eee0e955e6503f00\": container with ID starting with 25de6eb9b21dde21f7a1f94cb9adaf868d5d5805edbbb052eee0e955e6503f00 not found: ID does not exist" containerID="25de6eb9b21dde21f7a1f94cb9adaf868d5d5805edbbb052eee0e955e6503f00"
Sep 09 15:45:05.369905 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:05.369853    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25de6eb9b21dde21f7a1f94cb9adaf868d5d5805edbbb052eee0e955e6503f00"} err="failed to get container status \"25de6eb9b21dde21f7a1f94cb9adaf868d5d5805edbbb052eee0e955e6503f00\": rpc error: code = NotFound desc = could not find container \"25de6eb9b21dde21f7a1f94cb9adaf868d5d5805edbbb052eee0e955e6503f00\": container with ID starting with 25de6eb9b21dde21f7a1f94cb9adaf868d5d5805edbbb052eee0e955e6503f00 not found: ID does not exist"
Sep 09 15:45:05.385698 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:05.385647    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-1e15c8/distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg" podStartSLOduration=2.385630006 podStartE2EDuration="2.385630006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:45:05.385431095 +0000 UTC m=+1470.984325741" watchObservedRunningTime="2026-09-09 15:45:05.385630006 +0000 UTC m=+1470.984524649"
Sep 09 15:45:05.423549 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:05.423517    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-57b579-58954c5b5d-dvsnc"]
Sep 09 15:45:05.435361 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:05.435334    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-57b579-58954c5b5d-dvsnc"]
Sep 09 15:45:05.457098 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:05.457069    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-57b579-7994b7cc87-fgscr"]
Sep 09 15:45:05.479234 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:05.479207    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-57b579-7994b7cc87-fgscr"]
Sep 09 15:45:06.963298 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:06.963256    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="015c0cd8-414e-4d52-aeee-c64c1588b315" path="/var/lib/kubelet/pods/015c0cd8-414e-4d52-aeee-c64c1588b315/volumes"
Sep 09 15:45:06.963870 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:06.963851    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d33f110-3b40-4098-8075-2a42dba9c751" path="/var/lib/kubelet/pods/9d33f110-3b40-4098-8075-2a42dba9c751/volumes"
Sep 09 15:45:13.989918 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:13.989880    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1e15c8/distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg"
Sep 09 15:45:13.990408 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:13.989934    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-1e15c8/distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg"
Sep 09 15:45:14.002966 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:14.002932    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-1e15c8/distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg"
Sep 09 15:45:14.420077 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:14.420045    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1e15c8/distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg"
Sep 09 15:45:34.986537 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:34.986503    2553 scope.go:117] "RemoveContainer" containerID="8736831e8494a4fb58474fbbef594ce62f7395faff9a1e7b8001705d80ca7df1"
Sep 09 15:45:47.547657 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:47.547621    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-267962-vp7ll"]
Sep 09 15:45:47.548137 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:47.548052    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d33f110-3b40-4098-8075-2a42dba9c751" containerName="payload-processing"
Sep 09 15:45:47.548137 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:47.548065    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d33f110-3b40-4098-8075-2a42dba9c751" containerName="payload-processing"
Sep 09 15:45:47.548137 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:47.548094    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="015c0cd8-414e-4d52-aeee-c64c1588b315" containerName="payload-pre-processing"
Sep 09 15:45:47.548137 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:47.548101    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="015c0cd8-414e-4d52-aeee-c64c1588b315" containerName="payload-pre-processing"
Sep 09 15:45:47.548274 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:47.548164    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d33f110-3b40-4098-8075-2a42dba9c751" containerName="payload-processing"
Sep 09 15:45:47.548274 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:47.548179    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="015c0cd8-414e-4d52-aeee-c64c1588b315" containerName="payload-pre-processing"
Sep 09 15:45:47.551249 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:47.551224    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-267962-vp7ll"
Sep 09 15:45:47.554114 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:47.554089    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2zlsf\""
Sep 09 15:45:47.554114 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:47.554105    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 15:45:47.560222 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:47.560193    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-267962-vp7ll"]
Sep 09 15:45:47.627392 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:47.627355    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dad2aad9-3f9e-4f0d-b6fd-78fac4fd1586-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-267962-vp7ll\" (UID: \"dad2aad9-3f9e-4f0d-b6fd-78fac4fd1586\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-267962-vp7ll"
Sep 09 15:45:47.729198 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:47.729152    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dad2aad9-3f9e-4f0d-b6fd-78fac4fd1586-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-267962-vp7ll\" (UID: \"dad2aad9-3f9e-4f0d-b6fd-78fac4fd1586\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-267962-vp7ll"
Sep 09 15:45:47.730079 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:47.730046    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dad2aad9-3f9e-4f0d-b6fd-78fac4fd1586-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-267962-vp7ll\" (UID: \"dad2aad9-3f9e-4f0d-b6fd-78fac4fd1586\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-267962-vp7ll"
Sep 09 15:45:47.863321 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:47.863226    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-267962-vp7ll"
Sep 09 15:45:48.141354 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:48.141325    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-267962-vp7ll"]
Sep 09 15:45:48.143673 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:45:48.143646    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddad2aad9_3f9e_4f0d_b6fd_78fac4fd1586.slice/crio-1911dc75088b3dba2d85f21f9ffba716977bf2f8c233faa5d2aab1dddacc54ef WatchSource:0}: Error finding container 1911dc75088b3dba2d85f21f9ffba716977bf2f8c233faa5d2aab1dddacc54ef: Status 404 returned error can't find the container with id 1911dc75088b3dba2d85f21f9ffba716977bf2f8c233faa5d2aab1dddacc54ef
Sep 09 15:45:48.607841 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:48.607804    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-267962-vp7ll" event={"ID":"dad2aad9-3f9e-4f0d-b6fd-78fac4fd1586","Type":"ContainerStarted","Data":"fc0e1876aa5f5b4eb07c2a97af807623935f322d989e0458334833e1e24a7e42"}
Sep 09 15:45:48.608268 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:48.607866    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-267962-vp7ll" event={"ID":"dad2aad9-3f9e-4f0d-b6fd-78fac4fd1586","Type":"ContainerStarted","Data":"1911dc75088b3dba2d85f21f9ffba716977bf2f8c233faa5d2aab1dddacc54ef"}
Sep 09 15:45:48.623782 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:48.623721    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-267962-vp7ll" podStartSLOduration=1.623705562 podStartE2EDuration="1.623705562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:45:48.621705415 +0000 UTC m=+1514.220600060" watchObservedRunningTime="2026-09-09 15:45:48.623705562 +0000 UTC m=+1514.222600205"
Sep 09 15:45:49.615358 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:49.615320    2553 generic.go:356] "Generic (PLEG): container finished" podID="dad2aad9-3f9e-4f0d-b6fd-78fac4fd1586" containerID="fc0e1876aa5f5b4eb07c2a97af807623935f322d989e0458334833e1e24a7e42" exitCode=0
Sep 09 15:45:49.615833 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:49.615405    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-267962-vp7ll" event={"ID":"dad2aad9-3f9e-4f0d-b6fd-78fac4fd1586","Type":"ContainerDied","Data":"fc0e1876aa5f5b4eb07c2a97af807623935f322d989e0458334833e1e24a7e42"}
Sep 09 15:45:50.761814 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:50.761788    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-267962-vp7ll"
Sep 09 15:45:50.863080 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:50.863048    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dad2aad9-3f9e-4f0d-b6fd-78fac4fd1586-maas-api-service-ca\") pod \"dad2aad9-3f9e-4f0d-b6fd-78fac4fd1586\" (UID: \"dad2aad9-3f9e-4f0d-b6fd-78fac4fd1586\") "
Sep 09 15:45:50.863391 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:50.863369    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dad2aad9-3f9e-4f0d-b6fd-78fac4fd1586-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dad2aad9-3f9e-4f0d-b6fd-78fac4fd1586" (UID: "dad2aad9-3f9e-4f0d-b6fd-78fac4fd1586"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:45:50.963838 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:50.963806    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dad2aad9-3f9e-4f0d-b6fd-78fac4fd1586-maas-api-service-ca\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:45:51.628279 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:51.628247    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-267962-vp7ll" event={"ID":"dad2aad9-3f9e-4f0d-b6fd-78fac4fd1586","Type":"ContainerDied","Data":"1911dc75088b3dba2d85f21f9ffba716977bf2f8c233faa5d2aab1dddacc54ef"}
Sep 09 15:45:51.628431 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:51.628302    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1911dc75088b3dba2d85f21f9ffba716977bf2f8c233faa5d2aab1dddacc54ef"
Sep 09 15:45:51.628431 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:51.628261    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-267962-vp7ll"
Sep 09 15:45:57.999184 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:57.998974    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-267962-29816145-bg6vw"]
Sep 09 15:45:58.002522 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:58.002487    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-267962-29816145-bg6vw"]
Sep 09 15:45:58.028639 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:58.028606    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-267962-796f8bdb8b-pxs9m"]
Sep 09 15:45:58.029316 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:58.028834    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-267962-796f8bdb8b-pxs9m" podUID="c55048a7-6b0b-4b63-99c8-cb695ed7217f" containerName="payload-pre-processing" containerID="cri-o://97e09bab7be1a41efe33b3625f358b1087640dec7634b3cd8b473f45a114a292" gracePeriod=30
Sep 09 15:45:58.173661 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:58.173616    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-267962-vp7ll"]
Sep 09 15:45:58.177080 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:58.177051    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-267962-vp7ll"]
Sep 09 15:45:58.296703 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:58.296677    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-267962-796f8bdb8b-pxs9m"
Sep 09 15:45:58.440771 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:58.440705    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c55048a7-6b0b-4b63-99c8-cb695ed7217f-plugins-config-volume\") pod \"c55048a7-6b0b-4b63-99c8-cb695ed7217f\" (UID: \"c55048a7-6b0b-4b63-99c8-cb695ed7217f\") "
Sep 09 15:45:58.440968 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:58.440853    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljlc7\" (UniqueName: \"kubernetes.io/projected/c55048a7-6b0b-4b63-99c8-cb695ed7217f-kube-api-access-ljlc7\") pod \"c55048a7-6b0b-4b63-99c8-cb695ed7217f\" (UID: \"c55048a7-6b0b-4b63-99c8-cb695ed7217f\") "
Sep 09 15:45:58.441146 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:58.441122    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c55048a7-6b0b-4b63-99c8-cb695ed7217f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c55048a7-6b0b-4b63-99c8-cb695ed7217f" (UID: "c55048a7-6b0b-4b63-99c8-cb695ed7217f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:45:58.442894 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:58.442869    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c55048a7-6b0b-4b63-99c8-cb695ed7217f-kube-api-access-ljlc7" (OuterVolumeSpecName: "kube-api-access-ljlc7") pod "c55048a7-6b0b-4b63-99c8-cb695ed7217f" (UID: "c55048a7-6b0b-4b63-99c8-cb695ed7217f"). InnerVolumeSpecName "kube-api-access-ljlc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:45:58.541962 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:58.541849    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljlc7\" (UniqueName: \"kubernetes.io/projected/c55048a7-6b0b-4b63-99c8-cb695ed7217f-kube-api-access-ljlc7\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:45:58.541962 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:58.541891    2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c55048a7-6b0b-4b63-99c8-cb695ed7217f-plugins-config-volume\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:45:58.670930 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:58.670891    2553 generic.go:356] "Generic (PLEG): container finished" podID="c55048a7-6b0b-4b63-99c8-cb695ed7217f" containerID="97e09bab7be1a41efe33b3625f358b1087640dec7634b3cd8b473f45a114a292" exitCode=0
Sep 09 15:45:58.671115 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:58.670946    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-267962-796f8bdb8b-pxs9m"
Sep 09 15:45:58.671115 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:58.670967    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-267962-796f8bdb8b-pxs9m" event={"ID":"c55048a7-6b0b-4b63-99c8-cb695ed7217f","Type":"ContainerDied","Data":"97e09bab7be1a41efe33b3625f358b1087640dec7634b3cd8b473f45a114a292"}
Sep 09 15:45:58.671115 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:58.671014    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-267962-796f8bdb8b-pxs9m" event={"ID":"c55048a7-6b0b-4b63-99c8-cb695ed7217f","Type":"ContainerDied","Data":"e721d29be672442355318c4dd239bd9d378a24025481e134cf236f432694c183"}
Sep 09 15:45:58.671115 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:58.671034    2553 scope.go:117] "RemoveContainer" containerID="97e09bab7be1a41efe33b3625f358b1087640dec7634b3cd8b473f45a114a292"
Sep 09 15:45:58.680672 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:58.680653    2553 scope.go:117] "RemoveContainer" containerID="97e09bab7be1a41efe33b3625f358b1087640dec7634b3cd8b473f45a114a292"
Sep 09 15:45:58.680975 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:45:58.680957    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97e09bab7be1a41efe33b3625f358b1087640dec7634b3cd8b473f45a114a292\": container with ID starting with 97e09bab7be1a41efe33b3625f358b1087640dec7634b3cd8b473f45a114a292 not found: ID does not exist" containerID="97e09bab7be1a41efe33b3625f358b1087640dec7634b3cd8b473f45a114a292"
Sep 09 15:45:58.681026 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:58.680985    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97e09bab7be1a41efe33b3625f358b1087640dec7634b3cd8b473f45a114a292"} err="failed to get container status \"97e09bab7be1a41efe33b3625f358b1087640dec7634b3cd8b473f45a114a292\": rpc error: code = NotFound desc = could not find container \"97e09bab7be1a41efe33b3625f358b1087640dec7634b3cd8b473f45a114a292\": container with ID starting with 97e09bab7be1a41efe33b3625f358b1087640dec7634b3cd8b473f45a114a292 not found: ID does not exist"
Sep 09 15:45:58.693010 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:58.692981    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-267962-796f8bdb8b-pxs9m"]
Sep 09 15:45:58.697167 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:58.697144    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-267962-796f8bdb8b-pxs9m"]
Sep 09 15:45:58.962551 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:58.962512    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37439468-c16a-49ad-af2d-724f3fc4c9ff" path="/var/lib/kubelet/pods/37439468-c16a-49ad-af2d-724f3fc4c9ff/volumes"
Sep 09 15:45:58.962881 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:58.962867    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c55048a7-6b0b-4b63-99c8-cb695ed7217f" path="/var/lib/kubelet/pods/c55048a7-6b0b-4b63-99c8-cb695ed7217f/volumes"
Sep 09 15:45:58.963185 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:45:58.963173    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dad2aad9-3f9e-4f0d-b6fd-78fac4fd1586" path="/var/lib/kubelet/pods/dad2aad9-3f9e-4f0d-b6fd-78fac4fd1586/volumes"
Sep 09 15:46:03.565673 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:03.565633    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"]
Sep 09 15:46:03.566559 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:03.566525    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm" podUID="5485277a-06c9-432d-ab56-dcfd6cbec788" containerName="istio-proxy" containerID="cri-o://dad9932d50f94faf06a7650f76ac883aa86d9d7c4730b16a57bb1f0d2ac34ff3" gracePeriod=30
Sep 09 15:46:04.837030 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:04.836987    2553 patch_prober.go:28] interesting pod/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 15:46:04.837481 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:04.837056    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm" podUID="5485277a-06c9-432d-ab56-dcfd6cbec788" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 15:46:08.729546 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:08.729503    2553 generic.go:356] "Generic (PLEG): container finished" podID="5485277a-06c9-432d-ab56-dcfd6cbec788" containerID="dad9932d50f94faf06a7650f76ac883aa86d9d7c4730b16a57bb1f0d2ac34ff3" exitCode=0
Sep 09 15:46:08.729945 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:08.729580    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm" event={"ID":"5485277a-06c9-432d-ab56-dcfd6cbec788","Type":"ContainerDied","Data":"dad9932d50f94faf06a7650f76ac883aa86d9d7c4730b16a57bb1f0d2ac34ff3"}
Sep 09 15:46:08.833236 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:08.833161    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"
Sep 09 15:46:08.943836 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:08.943798    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4b4gj\" (UniqueName: \"kubernetes.io/projected/5485277a-06c9-432d-ab56-dcfd6cbec788-kube-api-access-4b4gj\") pod \"5485277a-06c9-432d-ab56-dcfd6cbec788\" (UID: \"5485277a-06c9-432d-ab56-dcfd6cbec788\") "
Sep 09 15:46:08.944009 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:08.943846    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5485277a-06c9-432d-ab56-dcfd6cbec788-istio-podinfo\") pod \"5485277a-06c9-432d-ab56-dcfd6cbec788\" (UID: \"5485277a-06c9-432d-ab56-dcfd6cbec788\") "
Sep 09 15:46:08.944009 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:08.943930    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5485277a-06c9-432d-ab56-dcfd6cbec788-istio-envoy\") pod \"5485277a-06c9-432d-ab56-dcfd6cbec788\" (UID: \"5485277a-06c9-432d-ab56-dcfd6cbec788\") "
Sep 09 15:46:08.944009 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:08.943973    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5485277a-06c9-432d-ab56-dcfd6cbec788-credential-socket\") pod \"5485277a-06c9-432d-ab56-dcfd6cbec788\" (UID: \"5485277a-06c9-432d-ab56-dcfd6cbec788\") "
Sep 09 15:46:08.944154 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:08.944095    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5485277a-06c9-432d-ab56-dcfd6cbec788-istio-token\") pod \"5485277a-06c9-432d-ab56-dcfd6cbec788\" (UID: \"5485277a-06c9-432d-ab56-dcfd6cbec788\") "
Sep 09 15:46:08.944215 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:08.944175    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5485277a-06c9-432d-ab56-dcfd6cbec788-workload-certs\") pod \"5485277a-06c9-432d-ab56-dcfd6cbec788\" (UID: \"5485277a-06c9-432d-ab56-dcfd6cbec788\") "
Sep 09 15:46:08.944270 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:08.944234    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5485277a-06c9-432d-ab56-dcfd6cbec788-istiod-ca-cert\") pod \"5485277a-06c9-432d-ab56-dcfd6cbec788\" (UID: \"5485277a-06c9-432d-ab56-dcfd6cbec788\") "
Sep 09 15:46:08.944270 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:08.944252    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5485277a-06c9-432d-ab56-dcfd6cbec788-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5485277a-06c9-432d-ab56-dcfd6cbec788" (UID: "5485277a-06c9-432d-ab56-dcfd6cbec788"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:46:08.944376 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:08.944268    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5485277a-06c9-432d-ab56-dcfd6cbec788-istio-data\") pod \"5485277a-06c9-432d-ab56-dcfd6cbec788\" (UID: \"5485277a-06c9-432d-ab56-dcfd6cbec788\") "
Sep 09 15:46:08.944376 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:08.944331    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5485277a-06c9-432d-ab56-dcfd6cbec788-workload-socket\") pod \"5485277a-06c9-432d-ab56-dcfd6cbec788\" (UID: \"5485277a-06c9-432d-ab56-dcfd6cbec788\") "
Sep 09 15:46:08.944515 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:08.944480    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5485277a-06c9-432d-ab56-dcfd6cbec788-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5485277a-06c9-432d-ab56-dcfd6cbec788" (UID: "5485277a-06c9-432d-ab56-dcfd6cbec788"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:46:08.944515 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:08.944494    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5485277a-06c9-432d-ab56-dcfd6cbec788-istio-data" (OuterVolumeSpecName: "istio-data") pod "5485277a-06c9-432d-ab56-dcfd6cbec788" (UID: "5485277a-06c9-432d-ab56-dcfd6cbec788"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:46:08.944698 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:08.944676    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5485277a-06c9-432d-ab56-dcfd6cbec788-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5485277a-06c9-432d-ab56-dcfd6cbec788" (UID: "5485277a-06c9-432d-ab56-dcfd6cbec788"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:46:08.944864 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:08.944777    2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5485277a-06c9-432d-ab56-dcfd6cbec788-istio-data\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:46:08.944864 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:08.944705    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5485277a-06c9-432d-ab56-dcfd6cbec788-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5485277a-06c9-432d-ab56-dcfd6cbec788" (UID: "5485277a-06c9-432d-ab56-dcfd6cbec788"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:46:08.944864 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:08.944804    2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5485277a-06c9-432d-ab56-dcfd6cbec788-credential-socket\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:46:08.944864 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:08.944820    2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5485277a-06c9-432d-ab56-dcfd6cbec788-workload-certs\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:46:08.946173 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:08.946147    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5485277a-06c9-432d-ab56-dcfd6cbec788-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5485277a-06c9-432d-ab56-dcfd6cbec788" (UID: "5485277a-06c9-432d-ab56-dcfd6cbec788"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:46:08.946386 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:08.946368    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5485277a-06c9-432d-ab56-dcfd6cbec788-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5485277a-06c9-432d-ab56-dcfd6cbec788" (UID: "5485277a-06c9-432d-ab56-dcfd6cbec788"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 15:46:08.946676 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:08.946649    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5485277a-06c9-432d-ab56-dcfd6cbec788-kube-api-access-4b4gj" (OuterVolumeSpecName: "kube-api-access-4b4gj") pod "5485277a-06c9-432d-ab56-dcfd6cbec788" (UID: "5485277a-06c9-432d-ab56-dcfd6cbec788"). InnerVolumeSpecName "kube-api-access-4b4gj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:46:08.946676 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:08.946658    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5485277a-06c9-432d-ab56-dcfd6cbec788-istio-token" (OuterVolumeSpecName: "istio-token") pod "5485277a-06c9-432d-ab56-dcfd6cbec788" (UID: "5485277a-06c9-432d-ab56-dcfd6cbec788"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:46:09.046239 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:09.046206    2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5485277a-06c9-432d-ab56-dcfd6cbec788-istiod-ca-cert\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:46:09.046239 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:09.046237    2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5485277a-06c9-432d-ab56-dcfd6cbec788-workload-socket\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:46:09.046410 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:09.046249    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b4gj\" (UniqueName: \"kubernetes.io/projected/5485277a-06c9-432d-ab56-dcfd6cbec788-kube-api-access-4b4gj\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:46:09.046410 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:09.046260    2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5485277a-06c9-432d-ab56-dcfd6cbec788-istio-podinfo\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:46:09.046410 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:09.046271    2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5485277a-06c9-432d-ab56-dcfd6cbec788-istio-envoy\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:46:09.046410 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:09.046280    2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5485277a-06c9-432d-ab56-dcfd6cbec788-istio-token\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:46:09.736950 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:09.736919    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"
Sep 09 15:46:09.736950 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:09.736937    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm" event={"ID":"5485277a-06c9-432d-ab56-dcfd6cbec788","Type":"ContainerDied","Data":"8ba3e2a9bd801b8e1b63c2f02397d879b2944f7057c1f6569ddc1331718fe6b2"}
Sep 09 15:46:09.737481 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:09.737007    2553 scope.go:117] "RemoveContainer" containerID="dad9932d50f94faf06a7650f76ac883aa86d9d7c4730b16a57bb1f0d2ac34ff3"
Sep 09 15:46:09.757182 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:09.757155    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"]
Sep 09 15:46:09.761267 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:09.761244    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-a0938b-openshift-default-7f7799b54b-886dm"]
Sep 09 15:46:10.966950 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:10.963571    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5485277a-06c9-432d-ab56-dcfd6cbec788" path="/var/lib/kubelet/pods/5485277a-06c9-432d-ab56-dcfd6cbec788/volumes"
Sep 09 15:46:35.016236 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:35.016207    2553 scope.go:117] "RemoveContainer" containerID="0495d7a6fb15b99dc903a318ed01a3dbd22a692808e0eefff16abe5acd8f653c"
Sep 09 15:46:35.025232 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:35.025206    2553 scope.go:117] "RemoveContainer" containerID="1d94cf632fbc730193ba53ab27a4699fc904bb66222a5f8b4e521ccc53ec5e29"
Sep 09 15:46:35.033361 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:46:35.033336    2553 scope.go:117] "RemoveContainer" containerID="4b615a2a1859f58ba1901f651a7efe5547be330094e5da096162f9c59df577b8"
Sep 09 15:47:01.938348 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:01.938310    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86"]
Sep 09 15:47:01.938937 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:01.938917    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5485277a-06c9-432d-ab56-dcfd6cbec788" containerName="istio-proxy"
Sep 09 15:47:01.938937 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:01.938936    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="5485277a-06c9-432d-ab56-dcfd6cbec788" containerName="istio-proxy"
Sep 09 15:47:01.939083 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:01.938954    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dad2aad9-3f9e-4f0d-b6fd-78fac4fd1586" containerName="revoke-keys"
Sep 09 15:47:01.939083 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:01.938962    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="dad2aad9-3f9e-4f0d-b6fd-78fac4fd1586" containerName="revoke-keys"
Sep 09 15:47:01.939083 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:01.938992    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c55048a7-6b0b-4b63-99c8-cb695ed7217f" containerName="payload-pre-processing"
Sep 09 15:47:01.939083 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:01.939003    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c55048a7-6b0b-4b63-99c8-cb695ed7217f" containerName="payload-pre-processing"
Sep 09 15:47:01.939292 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:01.939093    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="5485277a-06c9-432d-ab56-dcfd6cbec788" containerName="istio-proxy"
Sep 09 15:47:01.939292 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:01.939104    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="dad2aad9-3f9e-4f0d-b6fd-78fac4fd1586" containerName="revoke-keys"
Sep 09 15:47:01.939292 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:01.939112    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c55048a7-6b0b-4b63-99c8-cb695ed7217f" containerName="payload-pre-processing"
Sep 09 15:47:01.942619 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:01.942592    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1e15c8/unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86"
Sep 09 15:47:01.945220 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:01.945190    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1e15c8\"/\"unconfigured-w4-1e15c8-kserve-self-signed-certs\""
Sep 09 15:47:01.949426 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:01.949400    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86"]
Sep 09 15:47:02.079684 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:02.079644    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7eeea333-89f5-4c49-bb04-6b860ef79ad8-model-cache\") pod \"unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86\" (UID: \"7eeea333-89f5-4c49-bb04-6b860ef79ad8\") " pod="e2e-models-e2e-worker-w4-1e15c8/unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86"
Sep 09 15:47:02.079909 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:02.079702    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7eeea333-89f5-4c49-bb04-6b860ef79ad8-home\") pod \"unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86\" (UID: \"7eeea333-89f5-4c49-bb04-6b860ef79ad8\") " pod="e2e-models-e2e-worker-w4-1e15c8/unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86"
Sep 09 15:47:02.079909 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:02.079724    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7eeea333-89f5-4c49-bb04-6b860ef79ad8-dshm\") pod \"unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86\" (UID: \"7eeea333-89f5-4c49-bb04-6b860ef79ad8\") " pod="e2e-models-e2e-worker-w4-1e15c8/unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86"
Sep 09 15:47:02.086070 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:02.079837    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fsz8\" (UniqueName: \"kubernetes.io/projected/7eeea333-89f5-4c49-bb04-6b860ef79ad8-kube-api-access-2fsz8\") pod \"unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86\" (UID: \"7eeea333-89f5-4c49-bb04-6b860ef79ad8\") " pod="e2e-models-e2e-worker-w4-1e15c8/unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86"
Sep 09 15:47:02.086070 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:02.080294    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7eeea333-89f5-4c49-bb04-6b860ef79ad8-tls-certs\") pod \"unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86\" (UID: \"7eeea333-89f5-4c49-bb04-6b860ef79ad8\") " pod="e2e-models-e2e-worker-w4-1e15c8/unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86"
Sep 09 15:47:02.086070 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:02.080412    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7eeea333-89f5-4c49-bb04-6b860ef79ad8-tmp-dir\") pod \"unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86\" (UID: \"7eeea333-89f5-4c49-bb04-6b860ef79ad8\") " pod="e2e-models-e2e-worker-w4-1e15c8/unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86"
Sep 09 15:47:02.181713 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:02.181654    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7eeea333-89f5-4c49-bb04-6b860ef79ad8-tmp-dir\") pod \"unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86\" (UID: \"7eeea333-89f5-4c49-bb04-6b860ef79ad8\") " pod="e2e-models-e2e-worker-w4-1e15c8/unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86"
Sep 09 15:47:02.181713 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:02.181720    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7eeea333-89f5-4c49-bb04-6b860ef79ad8-model-cache\") pod \"unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86\" (UID: \"7eeea333-89f5-4c49-bb04-6b860ef79ad8\") " pod="e2e-models-e2e-worker-w4-1e15c8/unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86"
Sep 09 15:47:02.182009 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:02.181789    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7eeea333-89f5-4c49-bb04-6b860ef79ad8-home\") pod \"unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86\" (UID: \"7eeea333-89f5-4c49-bb04-6b860ef79ad8\") " pod="e2e-models-e2e-worker-w4-1e15c8/unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86"
Sep 09 15:47:02.182009 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:02.181812    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7eeea333-89f5-4c49-bb04-6b860ef79ad8-dshm\") pod \"unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86\" (UID: \"7eeea333-89f5-4c49-bb04-6b860ef79ad8\") " pod="e2e-models-e2e-worker-w4-1e15c8/unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86"
Sep 09 15:47:02.182009 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:02.181852    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fsz8\" (UniqueName: \"kubernetes.io/projected/7eeea333-89f5-4c49-bb04-6b860ef79ad8-kube-api-access-2fsz8\") pod \"unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86\" (UID: \"7eeea333-89f5-4c49-bb04-6b860ef79ad8\") " pod="e2e-models-e2e-worker-w4-1e15c8/unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86"
Sep 09 15:47:02.182009 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:02.181880    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7eeea333-89f5-4c49-bb04-6b860ef79ad8-tls-certs\") pod \"unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86\" (UID: \"7eeea333-89f5-4c49-bb04-6b860ef79ad8\") " pod="e2e-models-e2e-worker-w4-1e15c8/unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86"
Sep 09 15:47:02.182510 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:02.182189    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7eeea333-89f5-4c49-bb04-6b860ef79ad8-tmp-dir\") pod \"unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86\" (UID: \"7eeea333-89f5-4c49-bb04-6b860ef79ad8\") " pod="e2e-models-e2e-worker-w4-1e15c8/unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86"
Sep 09 15:47:02.182510 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:02.182221    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7eeea333-89f5-4c49-bb04-6b860ef79ad8-model-cache\") pod \"unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86\" (UID: \"7eeea333-89f5-4c49-bb04-6b860ef79ad8\") " pod="e2e-models-e2e-worker-w4-1e15c8/unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86"
Sep 09 15:47:02.182510 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:02.182272    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7eeea333-89f5-4c49-bb04-6b860ef79ad8-home\") pod \"unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86\" (UID: \"7eeea333-89f5-4c49-bb04-6b860ef79ad8\") " pod="e2e-models-e2e-worker-w4-1e15c8/unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86"
Sep 09 15:47:02.184256 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:02.184234    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7eeea333-89f5-4c49-bb04-6b860ef79ad8-dshm\") pod \"unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86\" (UID: \"7eeea333-89f5-4c49-bb04-6b860ef79ad8\") " pod="e2e-models-e2e-worker-w4-1e15c8/unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86"
Sep 09 15:47:02.184496 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:02.184473    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7eeea333-89f5-4c49-bb04-6b860ef79ad8-tls-certs\") pod \"unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86\" (UID: \"7eeea333-89f5-4c49-bb04-6b860ef79ad8\") " pod="e2e-models-e2e-worker-w4-1e15c8/unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86"
Sep 09 15:47:02.190685 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:02.190625    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fsz8\" (UniqueName: \"kubernetes.io/projected/7eeea333-89f5-4c49-bb04-6b860ef79ad8-kube-api-access-2fsz8\") pod \"unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86\" (UID: \"7eeea333-89f5-4c49-bb04-6b860ef79ad8\") " pod="e2e-models-e2e-worker-w4-1e15c8/unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86"
Sep 09 15:47:02.255800 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:02.255762    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1e15c8/unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86"
Sep 09 15:47:02.527558 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:02.527528    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86"]
Sep 09 15:47:02.529038 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:47:02.529008    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7eeea333_89f5_4c49_bb04_6b860ef79ad8.slice/crio-2b35b6bd81dfe7510edd2bfdfcdb9359e6c887ca416b7bb5c2875b656b4efa60 WatchSource:0}: Error finding container 2b35b6bd81dfe7510edd2bfdfcdb9359e6c887ca416b7bb5c2875b656b4efa60: Status 404 returned error can't find the container with id 2b35b6bd81dfe7510edd2bfdfcdb9359e6c887ca416b7bb5c2875b656b4efa60
Sep 09 15:47:03.031014 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:03.030978    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1e15c8/unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86" event={"ID":"7eeea333-89f5-4c49-bb04-6b860ef79ad8","Type":"ContainerStarted","Data":"603fd235c570cca63eeaeef18ad554c448bf740c4a399f0a382bbf1534c6fbb2"}
Sep 09 15:47:03.031014 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:03.031023    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1e15c8/unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86" event={"ID":"7eeea333-89f5-4c49-bb04-6b860ef79ad8","Type":"ContainerStarted","Data":"2b35b6bd81dfe7510edd2bfdfcdb9359e6c887ca416b7bb5c2875b656b4efa60"}
Sep 09 15:47:03.050311 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:03.050255    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-1e15c8/unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86" podStartSLOduration=2.050237839 podStartE2EDuration="2.050237839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:47:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:47:03.048203394 +0000 UTC m=+1588.647098041" watchObservedRunningTime="2026-09-09 15:47:03.050237839 +0000 UTC m=+1588.649132482"
Sep 09 15:47:12.256276 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:12.256233    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1e15c8/unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86"
Sep 09 15:47:12.256799 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:12.256294    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-1e15c8/unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86"
Sep 09 15:47:12.269228 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:12.269198    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-1e15c8/unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86"
Sep 09 15:47:13.096789 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:13.096761    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1e15c8/unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86"
Sep 09 15:47:16.223651 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:16.223618    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-482dce-5bf7f8dc94-n77km"]
Sep 09 15:47:16.227195 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:16.227179    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-482dce-5bf7f8dc94-n77km"
Sep 09 15:47:16.231279 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:16.231253    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-482dce-dockercfg-n2rtx\""
Sep 09 15:47:16.231434 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:16.231387    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-482dce\""
Sep 09 15:47:16.242266 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:16.242242    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-482dce-5bf7f8dc94-n77km"]
Sep 09 15:47:16.252124 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:16.252088    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-482dce-5c5c4c564f-vl84m"]
Sep 09 15:47:16.255830 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:16.255810    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-482dce-5c5c4c564f-vl84m"
Sep 09 15:47:16.266990 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:16.266964    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-482dce-5c5c4c564f-vl84m"]
Sep 09 15:47:16.321545 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:16.321507    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxshp\" (UniqueName: \"kubernetes.io/projected/a5673d1e-abdd-494c-a43b-87e0c0d96f9b-kube-api-access-nxshp\") pod \"payload-pre-processing-e2e-shared-b-w3-482dce-5bf7f8dc94-n77km\" (UID: \"a5673d1e-abdd-494c-a43b-87e0c0d96f9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-482dce-5bf7f8dc94-n77km"
Sep 09 15:47:16.321852 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:16.321820    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc468\" (UniqueName: \"kubernetes.io/projected/cdbe03f8-06de-4b6e-8f5b-960b4ff8a797-kube-api-access-wc468\") pod \"payload-processing-e2e-shared-b-w3-482dce-5c5c4c564f-vl84m\" (UID: \"cdbe03f8-06de-4b6e-8f5b-960b4ff8a797\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-482dce-5c5c4c564f-vl84m"
Sep 09 15:47:16.321993 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:16.321865    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5673d1e-abdd-494c-a43b-87e0c0d96f9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-482dce-5bf7f8dc94-n77km\" (UID: \"a5673d1e-abdd-494c-a43b-87e0c0d96f9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-482dce-5bf7f8dc94-n77km"
Sep 09 15:47:16.321993 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:16.321916    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdbe03f8-06de-4b6e-8f5b-960b4ff8a797-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-482dce-5c5c4c564f-vl84m\" (UID: \"cdbe03f8-06de-4b6e-8f5b-960b4ff8a797\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-482dce-5c5c4c564f-vl84m"
Sep 09 15:47:16.422698 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:16.422645    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxshp\" (UniqueName: \"kubernetes.io/projected/a5673d1e-abdd-494c-a43b-87e0c0d96f9b-kube-api-access-nxshp\") pod \"payload-pre-processing-e2e-shared-b-w3-482dce-5bf7f8dc94-n77km\" (UID: \"a5673d1e-abdd-494c-a43b-87e0c0d96f9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-482dce-5bf7f8dc94-n77km"
Sep 09 15:47:16.422698 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:16.422703    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc468\" (UniqueName: \"kubernetes.io/projected/cdbe03f8-06de-4b6e-8f5b-960b4ff8a797-kube-api-access-wc468\") pod \"payload-processing-e2e-shared-b-w3-482dce-5c5c4c564f-vl84m\" (UID: \"cdbe03f8-06de-4b6e-8f5b-960b4ff8a797\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-482dce-5c5c4c564f-vl84m"
Sep 09 15:47:16.422973 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:16.422725    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5673d1e-abdd-494c-a43b-87e0c0d96f9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-482dce-5bf7f8dc94-n77km\" (UID: \"a5673d1e-abdd-494c-a43b-87e0c0d96f9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-482dce-5bf7f8dc94-n77km"
Sep 09 15:47:16.422973 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:16.422784    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdbe03f8-06de-4b6e-8f5b-960b4ff8a797-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-482dce-5c5c4c564f-vl84m\" (UID: \"cdbe03f8-06de-4b6e-8f5b-960b4ff8a797\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-482dce-5c5c4c564f-vl84m"
Sep 09 15:47:16.423529 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:16.423509    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdbe03f8-06de-4b6e-8f5b-960b4ff8a797-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-482dce-5c5c4c564f-vl84m\" (UID: \"cdbe03f8-06de-4b6e-8f5b-960b4ff8a797\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-482dce-5c5c4c564f-vl84m"
Sep 09 15:47:16.423574 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:16.423529    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5673d1e-abdd-494c-a43b-87e0c0d96f9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-482dce-5bf7f8dc94-n77km\" (UID: \"a5673d1e-abdd-494c-a43b-87e0c0d96f9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-482dce-5bf7f8dc94-n77km"
Sep 09 15:47:16.430874 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:16.430842    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxshp\" (UniqueName: \"kubernetes.io/projected/a5673d1e-abdd-494c-a43b-87e0c0d96f9b-kube-api-access-nxshp\") pod \"payload-pre-processing-e2e-shared-b-w3-482dce-5bf7f8dc94-n77km\" (UID: \"a5673d1e-abdd-494c-a43b-87e0c0d96f9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-482dce-5bf7f8dc94-n77km"
Sep 09 15:47:16.430874 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:16.430870    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc468\" (UniqueName: \"kubernetes.io/projected/cdbe03f8-06de-4b6e-8f5b-960b4ff8a797-kube-api-access-wc468\") pod \"payload-processing-e2e-shared-b-w3-482dce-5c5c4c564f-vl84m\" (UID: \"cdbe03f8-06de-4b6e-8f5b-960b4ff8a797\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-482dce-5c5c4c564f-vl84m"
Sep 09 15:47:16.538248 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:16.538155    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-482dce-5bf7f8dc94-n77km"
Sep 09 15:47:16.567837 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:16.567803    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-482dce-5c5c4c564f-vl84m"
Sep 09 15:47:16.813679 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:16.813651    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-482dce-5bf7f8dc94-n77km"]
Sep 09 15:47:16.815633 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:47:16.815609    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5673d1e_abdd_494c_a43b_87e0c0d96f9b.slice/crio-5043f50af32ab8861cef5babbf0ba25f7a683e946f86960df9a8d51b0e53ad43 WatchSource:0}: Error finding container 5043f50af32ab8861cef5babbf0ba25f7a683e946f86960df9a8d51b0e53ad43: Status 404 returned error can't find the container with id 5043f50af32ab8861cef5babbf0ba25f7a683e946f86960df9a8d51b0e53ad43
Sep 09 15:47:16.848325 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:16.848304    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-482dce-5c5c4c564f-vl84m"]
Sep 09 15:47:16.849892 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:47:16.849868    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdbe03f8_06de_4b6e_8f5b_960b4ff8a797.slice/crio-47aeb66b5fc947d1e2a8c18a4eff5078595edadc0a34d84624e5850e95629685 WatchSource:0}: Error finding container 47aeb66b5fc947d1e2a8c18a4eff5078595edadc0a34d84624e5850e95629685: Status 404 returned error can't find the container with id 47aeb66b5fc947d1e2a8c18a4eff5078595edadc0a34d84624e5850e95629685
Sep 09 15:47:17.108640 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:17.108538    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-482dce-5c5c4c564f-vl84m" event={"ID":"cdbe03f8-06de-4b6e-8f5b-960b4ff8a797","Type":"ContainerStarted","Data":"e475807116c278cf76a6a9a8ce18bf9dcac52e6fdb6dc7ddb9c2ec35065e54e3"}
Sep 09 15:47:17.108640 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:17.108605    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-482dce-5c5c4c564f-vl84m" event={"ID":"cdbe03f8-06de-4b6e-8f5b-960b4ff8a797","Type":"ContainerStarted","Data":"47aeb66b5fc947d1e2a8c18a4eff5078595edadc0a34d84624e5850e95629685"}
Sep 09 15:47:17.108640 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:17.108639    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-482dce-5c5c4c564f-vl84m"
Sep 09 15:47:17.110164 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:17.110140    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-482dce-5bf7f8dc94-n77km" event={"ID":"a5673d1e-abdd-494c-a43b-87e0c0d96f9b","Type":"ContainerStarted","Data":"6ad40de535c01a8b61fadf1a3b4e8de1fea8245fe1df54d0d2a215a3f8c91694"}
Sep 09 15:47:17.110298 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:17.110183    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-482dce-5bf7f8dc94-n77km" event={"ID":"a5673d1e-abdd-494c-a43b-87e0c0d96f9b","Type":"ContainerStarted","Data":"5043f50af32ab8861cef5babbf0ba25f7a683e946f86960df9a8d51b0e53ad43"}
Sep 09 15:47:17.110298 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:17.110214    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-482dce-5bf7f8dc94-n77km"
Sep 09 15:47:17.125670 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:17.125624    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-482dce-5c5c4c564f-vl84m" podStartSLOduration=1.125607264 podStartE2EDuration="1.125607264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:47:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:47:17.123968159 +0000 UTC m=+1602.722862802" watchObservedRunningTime="2026-09-09 15:47:17.125607264 +0000 UTC m=+1602.724502028"
Sep 09 15:47:17.140543 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:17.140492    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-482dce-5bf7f8dc94-n77km" podStartSLOduration=1.140477637 podStartE2EDuration="1.140477637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:47:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:47:17.139270239 +0000 UTC m=+1602.738164884" watchObservedRunningTime="2026-09-09 15:47:17.140477637 +0000 UTC m=+1602.739372280"
Sep 09 15:47:22.070547 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:22.070512    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aeb0f8ca-rjwf5"]
Sep 09 15:47:22.074158 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:22.074141    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aeb0f8ca-rjwf5"
Sep 09 15:47:22.076832 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:22.076809    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 15:47:22.076961 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:22.076867    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2zlsf\""
Sep 09 15:47:22.081138 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:22.081116    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aeb0f8ca-rjwf5"]
Sep 09 15:47:22.182309 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:22.182264    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b60d92-7b53-4d1b-83f5-52786cbacd4e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-aeb0f8ca-rjwf5\" (UID: \"79b60d92-7b53-4d1b-83f5-52786cbacd4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aeb0f8ca-rjwf5"
Sep 09 15:47:22.283558 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:22.283517    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b60d92-7b53-4d1b-83f5-52786cbacd4e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-aeb0f8ca-rjwf5\" (UID: \"79b60d92-7b53-4d1b-83f5-52786cbacd4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aeb0f8ca-rjwf5"
Sep 09 15:47:22.284259 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:22.284239    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b60d92-7b53-4d1b-83f5-52786cbacd4e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-aeb0f8ca-rjwf5\" (UID: \"79b60d92-7b53-4d1b-83f5-52786cbacd4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aeb0f8ca-rjwf5"
Sep 09 15:47:22.388121 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:22.388029    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aeb0f8ca-rjwf5"
Sep 09 15:47:22.661455 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:22.661423    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aeb0f8ca-rjwf5"]
Sep 09 15:47:22.663042 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:47:22.663006    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b60d92_7b53_4d1b_83f5_52786cbacd4e.slice/crio-29554331160da05fc8111a038095ec20c2d25cecdf19c23afd1e2ae7e235d728 WatchSource:0}: Error finding container 29554331160da05fc8111a038095ec20c2d25cecdf19c23afd1e2ae7e235d728: Status 404 returned error can't find the container with id 29554331160da05fc8111a038095ec20c2d25cecdf19c23afd1e2ae7e235d728
Sep 09 15:47:23.146191 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:23.146155    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aeb0f8ca-rjwf5" event={"ID":"79b60d92-7b53-4d1b-83f5-52786cbacd4e","Type":"ContainerStarted","Data":"6ce0a72cf0d8710448c4d876eeb2ea8a1f78a3e2e5347d7fb24bae36e4b0afc8"}
Sep 09 15:47:23.146590 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:23.146207    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aeb0f8ca-rjwf5" event={"ID":"79b60d92-7b53-4d1b-83f5-52786cbacd4e","Type":"ContainerStarted","Data":"29554331160da05fc8111a038095ec20c2d25cecdf19c23afd1e2ae7e235d728"}
Sep 09 15:47:23.161918 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:23.161868    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aeb0f8ca-rjwf5" podStartSLOduration=1.161851453 podStartE2EDuration="1.161851453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:47:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:47:23.16001659 +0000 UTC m=+1608.758911238" watchObservedRunningTime="2026-09-09 15:47:23.161851453 +0000 UTC m=+1608.760746097"
Sep 09 15:47:24.154385 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:24.154348    2553 generic.go:356] "Generic (PLEG): container finished" podID="79b60d92-7b53-4d1b-83f5-52786cbacd4e" containerID="6ce0a72cf0d8710448c4d876eeb2ea8a1f78a3e2e5347d7fb24bae36e4b0afc8" exitCode=0
Sep 09 15:47:24.154810 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:24.154433    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aeb0f8ca-rjwf5" event={"ID":"79b60d92-7b53-4d1b-83f5-52786cbacd4e","Type":"ContainerDied","Data":"6ce0a72cf0d8710448c4d876eeb2ea8a1f78a3e2e5347d7fb24bae36e4b0afc8"}
Sep 09 15:47:25.295424 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:25.295399    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aeb0f8ca-rjwf5"
Sep 09 15:47:25.414907 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:25.414818    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b60d92-7b53-4d1b-83f5-52786cbacd4e-maas-api-service-ca\") pod \"79b60d92-7b53-4d1b-83f5-52786cbacd4e\" (UID: \"79b60d92-7b53-4d1b-83f5-52786cbacd4e\") "
Sep 09 15:47:25.415190 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:25.415166    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79b60d92-7b53-4d1b-83f5-52786cbacd4e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "79b60d92-7b53-4d1b-83f5-52786cbacd4e" (UID: "79b60d92-7b53-4d1b-83f5-52786cbacd4e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:47:25.516221 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:25.516182    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b60d92-7b53-4d1b-83f5-52786cbacd4e-maas-api-service-ca\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:47:26.166891 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:26.166854    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aeb0f8ca-rjwf5" event={"ID":"79b60d92-7b53-4d1b-83f5-52786cbacd4e","Type":"ContainerDied","Data":"29554331160da05fc8111a038095ec20c2d25cecdf19c23afd1e2ae7e235d728"}
Sep 09 15:47:26.166891 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:26.166875    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aeb0f8ca-rjwf5"
Sep 09 15:47:26.167099 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:26.166928    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29554331160da05fc8111a038095ec20c2d25cecdf19c23afd1e2ae7e235d728"
Sep 09 15:47:28.118663 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:28.118575    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-482dce-5c5c4c564f-vl84m"
Sep 09 15:47:28.118663 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:28.118652    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-482dce-5bf7f8dc94-n77km"
Sep 09 15:47:37.628159 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:37.628124    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aeb0f8ca-29816145-7s6dx"]
Sep 09 15:47:37.632259 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:37.632229    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aeb0f8ca-29816145-7s6dx"]
Sep 09 15:47:37.658260 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:37.658226    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-aeb0f8ca-bfb9bf8f7-cq8m8"]
Sep 09 15:47:37.658489 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:37.658443    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-aeb0f8ca-bfb9bf8f7-cq8m8" podUID="ee78420e-573b-4064-8797-8b86d24fdfc0" containerName="payload-processing" containerID="cri-o://4e53c5f79974e6fe918cf706cbbcb130011cd60e7d12d086c044e5df14c2cc9d" gracePeriod=30
Sep 09 15:47:37.837180 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:37.837146    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aeb0f8ca-rjwf5"]
Sep 09 15:47:37.841581 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:37.840981    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aeb0f8ca-rjwf5"]
Sep 09 15:47:37.916336 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:37.916307    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-aeb0f8ca-bfb9bf8f7-cq8m8"
Sep 09 15:47:38.042849 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:38.042814    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee78420e-573b-4064-8797-8b86d24fdfc0-plugins-config-volume\") pod \"ee78420e-573b-4064-8797-8b86d24fdfc0\" (UID: \"ee78420e-573b-4064-8797-8b86d24fdfc0\") "
Sep 09 15:47:38.043024 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:38.042964    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crz4p\" (UniqueName: \"kubernetes.io/projected/ee78420e-573b-4064-8797-8b86d24fdfc0-kube-api-access-crz4p\") pod \"ee78420e-573b-4064-8797-8b86d24fdfc0\" (UID: \"ee78420e-573b-4064-8797-8b86d24fdfc0\") "
Sep 09 15:47:38.043194 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:38.043170    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee78420e-573b-4064-8797-8b86d24fdfc0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ee78420e-573b-4064-8797-8b86d24fdfc0" (UID: "ee78420e-573b-4064-8797-8b86d24fdfc0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:47:38.043344 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:38.043326    2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee78420e-573b-4064-8797-8b86d24fdfc0-plugins-config-volume\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:47:38.045080 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:38.045047    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee78420e-573b-4064-8797-8b86d24fdfc0-kube-api-access-crz4p" (OuterVolumeSpecName: "kube-api-access-crz4p") pod "ee78420e-573b-4064-8797-8b86d24fdfc0" (UID: "ee78420e-573b-4064-8797-8b86d24fdfc0"). InnerVolumeSpecName "kube-api-access-crz4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:47:38.143869 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:38.143832    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crz4p\" (UniqueName: \"kubernetes.io/projected/ee78420e-573b-4064-8797-8b86d24fdfc0-kube-api-access-crz4p\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:47:38.237658 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:38.237621    2553 generic.go:356] "Generic (PLEG): container finished" podID="ee78420e-573b-4064-8797-8b86d24fdfc0" containerID="4e53c5f79974e6fe918cf706cbbcb130011cd60e7d12d086c044e5df14c2cc9d" exitCode=0
Sep 09 15:47:38.237874 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:38.237669    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-aeb0f8ca-bfb9bf8f7-cq8m8"
Sep 09 15:47:38.237874 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:38.237707    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-aeb0f8ca-bfb9bf8f7-cq8m8" event={"ID":"ee78420e-573b-4064-8797-8b86d24fdfc0","Type":"ContainerDied","Data":"4e53c5f79974e6fe918cf706cbbcb130011cd60e7d12d086c044e5df14c2cc9d"}
Sep 09 15:47:38.237874 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:38.237792    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-aeb0f8ca-bfb9bf8f7-cq8m8" event={"ID":"ee78420e-573b-4064-8797-8b86d24fdfc0","Type":"ContainerDied","Data":"5e5800611dc374626f000c8ab646b491e351c939890059ee9ab27c3a44c3191c"}
Sep 09 15:47:38.237874 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:38.237827    2553 scope.go:117] "RemoveContainer" containerID="4e53c5f79974e6fe918cf706cbbcb130011cd60e7d12d086c044e5df14c2cc9d"
Sep 09 15:47:38.248379 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:38.248357    2553 scope.go:117] "RemoveContainer" containerID="4e53c5f79974e6fe918cf706cbbcb130011cd60e7d12d086c044e5df14c2cc9d"
Sep 09 15:47:38.248679 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:47:38.248659    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e53c5f79974e6fe918cf706cbbcb130011cd60e7d12d086c044e5df14c2cc9d\": container with ID starting with 4e53c5f79974e6fe918cf706cbbcb130011cd60e7d12d086c044e5df14c2cc9d not found: ID does not exist" containerID="4e53c5f79974e6fe918cf706cbbcb130011cd60e7d12d086c044e5df14c2cc9d"
Sep 09 15:47:38.248722 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:38.248693    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e53c5f79974e6fe918cf706cbbcb130011cd60e7d12d086c044e5df14c2cc9d"} err="failed to get container status \"4e53c5f79974e6fe918cf706cbbcb130011cd60e7d12d086c044e5df14c2cc9d\": rpc error: code = NotFound desc = could not find container \"4e53c5f79974e6fe918cf706cbbcb130011cd60e7d12d086c044e5df14c2cc9d\": container with ID starting with 4e53c5f79974e6fe918cf706cbbcb130011cd60e7d12d086c044e5df14c2cc9d not found: ID does not exist"
Sep 09 15:47:38.259374 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:38.259340    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-aeb0f8ca-bfb9bf8f7-cq8m8"]
Sep 09 15:47:38.263245 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:38.263221    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-aeb0f8ca-bfb9bf8f7-cq8m8"]
Sep 09 15:47:38.962420 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:38.962379    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="064b7c73-ea65-4dec-95b3-607237686388" path="/var/lib/kubelet/pods/064b7c73-ea65-4dec-95b3-607237686388/volumes"
Sep 09 15:47:38.962827 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:38.962698    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79b60d92-7b53-4d1b-83f5-52786cbacd4e" path="/var/lib/kubelet/pods/79b60d92-7b53-4d1b-83f5-52786cbacd4e/volumes"
Sep 09 15:47:38.963059 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:47:38.963045    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee78420e-573b-4064-8797-8b86d24fdfc0" path="/var/lib/kubelet/pods/ee78420e-573b-4064-8797-8b86d24fdfc0/volumes"
Sep 09 15:48:46.659577 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:46.659535    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-450a51/auth-test-model-450a51-kserve-86d7f44dcf-tlltv"]
Sep 09 15:48:46.660158 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:46.660137    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79b60d92-7b53-4d1b-83f5-52786cbacd4e" containerName="revoke-keys"
Sep 09 15:48:46.660202 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:46.660161    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="79b60d92-7b53-4d1b-83f5-52786cbacd4e" containerName="revoke-keys"
Sep 09 15:48:46.660202 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:46.660182    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee78420e-573b-4064-8797-8b86d24fdfc0" containerName="payload-processing"
Sep 09 15:48:46.660202 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:46.660191    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee78420e-573b-4064-8797-8b86d24fdfc0" containerName="payload-processing"
Sep 09 15:48:46.660311 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:46.660298    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee78420e-573b-4064-8797-8b86d24fdfc0" containerName="payload-processing"
Sep 09 15:48:46.660352 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:46.660312    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="79b60d92-7b53-4d1b-83f5-52786cbacd4e" containerName="revoke-keys"
Sep 09 15:48:46.663702 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:46.663686    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-450a51/auth-test-model-450a51-kserve-86d7f44dcf-tlltv"
Sep 09 15:48:46.666174 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:46.666148    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-450a51\"/\"kube-root-ca.crt\""
Sep 09 15:48:46.666174 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:46.666148    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-450a51\"/\"default-dockercfg-lqbr4\""
Sep 09 15:48:46.667437 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:46.667395    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-450a51\"/\"openshift-service-ca.crt\""
Sep 09 15:48:46.668998 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:46.668973    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-450a51\"/\"auth-test-model-450a51-kserve-self-signed-certs\""
Sep 09 15:48:46.678288 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:46.678258    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-450a51/auth-test-model-450a51-kserve-86d7f44dcf-tlltv"]
Sep 09 15:48:46.840450 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:46.840404    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s4jq\" (UniqueName: \"kubernetes.io/projected/c70ce445-aedc-4429-8beb-0f21a23c357b-kube-api-access-7s4jq\") pod \"auth-test-model-450a51-kserve-86d7f44dcf-tlltv\" (UID: \"c70ce445-aedc-4429-8beb-0f21a23c357b\") " pod="ai-tenant-e2e-shared-a-w3-450a51/auth-test-model-450a51-kserve-86d7f44dcf-tlltv"
Sep 09 15:48:46.840661 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:46.840584    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c70ce445-aedc-4429-8beb-0f21a23c357b-tls-certs\") pod \"auth-test-model-450a51-kserve-86d7f44dcf-tlltv\" (UID: \"c70ce445-aedc-4429-8beb-0f21a23c357b\") " pod="ai-tenant-e2e-shared-a-w3-450a51/auth-test-model-450a51-kserve-86d7f44dcf-tlltv"
Sep 09 15:48:46.840782 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:46.840709    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c70ce445-aedc-4429-8beb-0f21a23c357b-dshm\") pod \"auth-test-model-450a51-kserve-86d7f44dcf-tlltv\" (UID: \"c70ce445-aedc-4429-8beb-0f21a23c357b\") " pod="ai-tenant-e2e-shared-a-w3-450a51/auth-test-model-450a51-kserve-86d7f44dcf-tlltv"
Sep 09 15:48:46.840853 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:46.840816    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c70ce445-aedc-4429-8beb-0f21a23c357b-tmp-dir\") pod \"auth-test-model-450a51-kserve-86d7f44dcf-tlltv\" (UID: \"c70ce445-aedc-4429-8beb-0f21a23c357b\") " pod="ai-tenant-e2e-shared-a-w3-450a51/auth-test-model-450a51-kserve-86d7f44dcf-tlltv"
Sep 09 15:48:46.840910 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:46.840862    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c70ce445-aedc-4429-8beb-0f21a23c357b-home\") pod \"auth-test-model-450a51-kserve-86d7f44dcf-tlltv\" (UID: \"c70ce445-aedc-4429-8beb-0f21a23c357b\") " pod="ai-tenant-e2e-shared-a-w3-450a51/auth-test-model-450a51-kserve-86d7f44dcf-tlltv"
Sep 09 15:48:46.840964 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:46.840920    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c70ce445-aedc-4429-8beb-0f21a23c357b-model-cache\") pod \"auth-test-model-450a51-kserve-86d7f44dcf-tlltv\" (UID: \"c70ce445-aedc-4429-8beb-0f21a23c357b\") " pod="ai-tenant-e2e-shared-a-w3-450a51/auth-test-model-450a51-kserve-86d7f44dcf-tlltv"
Sep 09 15:48:46.942173 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:46.942128    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c70ce445-aedc-4429-8beb-0f21a23c357b-model-cache\") pod \"auth-test-model-450a51-kserve-86d7f44dcf-tlltv\" (UID: \"c70ce445-aedc-4429-8beb-0f21a23c357b\") " pod="ai-tenant-e2e-shared-a-w3-450a51/auth-test-model-450a51-kserve-86d7f44dcf-tlltv"
Sep 09 15:48:46.942336 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:46.942204    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s4jq\" (UniqueName: \"kubernetes.io/projected/c70ce445-aedc-4429-8beb-0f21a23c357b-kube-api-access-7s4jq\") pod \"auth-test-model-450a51-kserve-86d7f44dcf-tlltv\" (UID: \"c70ce445-aedc-4429-8beb-0f21a23c357b\") " pod="ai-tenant-e2e-shared-a-w3-450a51/auth-test-model-450a51-kserve-86d7f44dcf-tlltv"
Sep 09 15:48:46.942336 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:46.942236    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c70ce445-aedc-4429-8beb-0f21a23c357b-tls-certs\") pod \"auth-test-model-450a51-kserve-86d7f44dcf-tlltv\" (UID: \"c70ce445-aedc-4429-8beb-0f21a23c357b\") " pod="ai-tenant-e2e-shared-a-w3-450a51/auth-test-model-450a51-kserve-86d7f44dcf-tlltv"
Sep 09 15:48:46.942336 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:46.942313    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c70ce445-aedc-4429-8beb-0f21a23c357b-dshm\") pod \"auth-test-model-450a51-kserve-86d7f44dcf-tlltv\" (UID: \"c70ce445-aedc-4429-8beb-0f21a23c357b\") " pod="ai-tenant-e2e-shared-a-w3-450a51/auth-test-model-450a51-kserve-86d7f44dcf-tlltv"
Sep 09 15:48:46.942462 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:46.942413    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c70ce445-aedc-4429-8beb-0f21a23c357b-tmp-dir\") pod \"auth-test-model-450a51-kserve-86d7f44dcf-tlltv\" (UID: \"c70ce445-aedc-4429-8beb-0f21a23c357b\") " pod="ai-tenant-e2e-shared-a-w3-450a51/auth-test-model-450a51-kserve-86d7f44dcf-tlltv"
Sep 09 15:48:46.942501 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:46.942470    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c70ce445-aedc-4429-8beb-0f21a23c357b-home\") pod \"auth-test-model-450a51-kserve-86d7f44dcf-tlltv\" (UID: \"c70ce445-aedc-4429-8beb-0f21a23c357b\") " pod="ai-tenant-e2e-shared-a-w3-450a51/auth-test-model-450a51-kserve-86d7f44dcf-tlltv"
Sep 09 15:48:46.942619 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:46.942596    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c70ce445-aedc-4429-8beb-0f21a23c357b-model-cache\") pod \"auth-test-model-450a51-kserve-86d7f44dcf-tlltv\" (UID: \"c70ce445-aedc-4429-8beb-0f21a23c357b\") " pod="ai-tenant-e2e-shared-a-w3-450a51/auth-test-model-450a51-kserve-86d7f44dcf-tlltv"
Sep 09 15:48:46.942872 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:46.942842    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c70ce445-aedc-4429-8beb-0f21a23c357b-tmp-dir\") pod \"auth-test-model-450a51-kserve-86d7f44dcf-tlltv\" (UID: \"c70ce445-aedc-4429-8beb-0f21a23c357b\") " pod="ai-tenant-e2e-shared-a-w3-450a51/auth-test-model-450a51-kserve-86d7f44dcf-tlltv"
Sep 09 15:48:46.943012 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:46.942887    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c70ce445-aedc-4429-8beb-0f21a23c357b-home\") pod \"auth-test-model-450a51-kserve-86d7f44dcf-tlltv\" (UID: \"c70ce445-aedc-4429-8beb-0f21a23c357b\") " pod="ai-tenant-e2e-shared-a-w3-450a51/auth-test-model-450a51-kserve-86d7f44dcf-tlltv"
Sep 09 15:48:46.944572 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:46.944551    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c70ce445-aedc-4429-8beb-0f21a23c357b-dshm\") pod \"auth-test-model-450a51-kserve-86d7f44dcf-tlltv\" (UID: \"c70ce445-aedc-4429-8beb-0f21a23c357b\") " pod="ai-tenant-e2e-shared-a-w3-450a51/auth-test-model-450a51-kserve-86d7f44dcf-tlltv"
Sep 09 15:48:46.944890 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:46.944862    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c70ce445-aedc-4429-8beb-0f21a23c357b-tls-certs\") pod \"auth-test-model-450a51-kserve-86d7f44dcf-tlltv\" (UID: \"c70ce445-aedc-4429-8beb-0f21a23c357b\") " pod="ai-tenant-e2e-shared-a-w3-450a51/auth-test-model-450a51-kserve-86d7f44dcf-tlltv"
Sep 09 15:48:46.950257 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:46.950235    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s4jq\" (UniqueName: \"kubernetes.io/projected/c70ce445-aedc-4429-8beb-0f21a23c357b-kube-api-access-7s4jq\") pod \"auth-test-model-450a51-kserve-86d7f44dcf-tlltv\" (UID: \"c70ce445-aedc-4429-8beb-0f21a23c357b\") " pod="ai-tenant-e2e-shared-a-w3-450a51/auth-test-model-450a51-kserve-86d7f44dcf-tlltv"
Sep 09 15:48:46.983556 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:46.983521    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-450a51/auth-test-model-450a51-kserve-86d7f44dcf-tlltv"
Sep 09 15:48:47.256209 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:47.256183    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-450a51/auth-test-model-450a51-kserve-86d7f44dcf-tlltv"]
Sep 09 15:48:47.258394 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:48:47.258358    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc70ce445_aedc_4429_8beb_0f21a23c357b.slice/crio-20f9d94ed77727c0629939843e4ffb87557eb8fd84f852b67d02ac7416569f06 WatchSource:0}: Error finding container 20f9d94ed77727c0629939843e4ffb87557eb8fd84f852b67d02ac7416569f06: Status 404 returned error can't find the container with id 20f9d94ed77727c0629939843e4ffb87557eb8fd84f852b67d02ac7416569f06
Sep 09 15:48:47.637291 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:47.637206    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-450a51/auth-test-model-450a51-kserve-86d7f44dcf-tlltv" event={"ID":"c70ce445-aedc-4429-8beb-0f21a23c357b","Type":"ContainerStarted","Data":"8104f36df9ad1d14591d04cc9fc32e02d054e63419a09bf43b263226acea4109"}
Sep 09 15:48:47.637291 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:47.637250    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-450a51/auth-test-model-450a51-kserve-86d7f44dcf-tlltv" event={"ID":"c70ce445-aedc-4429-8beb-0f21a23c357b","Type":"ContainerStarted","Data":"20f9d94ed77727c0629939843e4ffb87557eb8fd84f852b67d02ac7416569f06"}
Sep 09 15:48:47.656148 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:47.656086    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-450a51/auth-test-model-450a51-kserve-86d7f44dcf-tlltv" podStartSLOduration=1.656071273 podStartE2EDuration="1.656071273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:48:47.653918634 +0000 UTC m=+1693.252813278" watchObservedRunningTime="2026-09-09 15:48:47.656071273 +0000 UTC m=+1693.254965913"
Sep 09 15:48:56.984140 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:56.984057    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-450a51/auth-test-model-450a51-kserve-86d7f44dcf-tlltv"
Sep 09 15:48:56.984140 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:56.984114    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-450a51/auth-test-model-450a51-kserve-86d7f44dcf-tlltv"
Sep 09 15:48:56.997224 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:56.997197    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-450a51/auth-test-model-450a51-kserve-86d7f44dcf-tlltv"
Sep 09 15:48:57.704233 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:48:57.704202    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-450a51/auth-test-model-450a51-kserve-86d7f44dcf-tlltv"
Sep 09 15:49:35.092646 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:35.092613    2553 scope.go:117] "RemoveContainer" containerID="ea770044cb6acb0d9352b5913b6504bc7de72e6be21939f46197e8d75247b081"
Sep 09 15:49:35.101598 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:35.101573    2553 scope.go:117] "RemoveContainer" containerID="5f556c512476fc2562c6c6df1add36b72bb3d469e2feabe9e2294d6212ceee16"
Sep 09 15:49:42.855210 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:42.855173    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-482dce/auth-test-model-482dce-kserve-67d5796c5f-zpknh"]
Sep 09 15:49:42.858797 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:42.858777    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-482dce/auth-test-model-482dce-kserve-67d5796c5f-zpknh"
Sep 09 15:49:42.862669 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:42.862642    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-482dce\"/\"auth-test-model-482dce-kserve-self-signed-certs\""
Sep 09 15:49:42.862898 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:42.862663    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-482dce\"/\"openshift-service-ca.crt\""
Sep 09 15:49:42.863060 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:42.862830    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-482dce\"/\"kube-root-ca.crt\""
Sep 09 15:49:42.863248 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:42.862912    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-482dce\"/\"default-dockercfg-95fv4\""
Sep 09 15:49:42.873394 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:42.873360    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-482dce/auth-test-model-482dce-kserve-67d5796c5f-zpknh"]
Sep 09 15:49:42.909570 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:42.909531    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb2a60ce-3718-4c59-8c72-7608c97b6027-home\") pod \"auth-test-model-482dce-kserve-67d5796c5f-zpknh\" (UID: \"cb2a60ce-3718-4c59-8c72-7608c97b6027\") " pod="ai-tenant-e2e-shared-b-w3-482dce/auth-test-model-482dce-kserve-67d5796c5f-zpknh"
Sep 09 15:49:42.909570 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:42.909574    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb2a60ce-3718-4c59-8c72-7608c97b6027-dshm\") pod \"auth-test-model-482dce-kserve-67d5796c5f-zpknh\" (UID: \"cb2a60ce-3718-4c59-8c72-7608c97b6027\") " pod="ai-tenant-e2e-shared-b-w3-482dce/auth-test-model-482dce-kserve-67d5796c5f-zpknh"
Sep 09 15:49:42.909812 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:42.909607    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb2a60ce-3718-4c59-8c72-7608c97b6027-tmp-dir\") pod \"auth-test-model-482dce-kserve-67d5796c5f-zpknh\" (UID: \"cb2a60ce-3718-4c59-8c72-7608c97b6027\") " pod="ai-tenant-e2e-shared-b-w3-482dce/auth-test-model-482dce-kserve-67d5796c5f-zpknh"
Sep 09 15:49:42.909812 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:42.909654    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb2a60ce-3718-4c59-8c72-7608c97b6027-tls-certs\") pod \"auth-test-model-482dce-kserve-67d5796c5f-zpknh\" (UID: \"cb2a60ce-3718-4c59-8c72-7608c97b6027\") " pod="ai-tenant-e2e-shared-b-w3-482dce/auth-test-model-482dce-kserve-67d5796c5f-zpknh"
Sep 09 15:49:42.909812 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:42.909674    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sb6r\" (UniqueName: \"kubernetes.io/projected/cb2a60ce-3718-4c59-8c72-7608c97b6027-kube-api-access-2sb6r\") pod \"auth-test-model-482dce-kserve-67d5796c5f-zpknh\" (UID: \"cb2a60ce-3718-4c59-8c72-7608c97b6027\") " pod="ai-tenant-e2e-shared-b-w3-482dce/auth-test-model-482dce-kserve-67d5796c5f-zpknh"
Sep 09 15:49:42.909812 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:42.909749    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb2a60ce-3718-4c59-8c72-7608c97b6027-model-cache\") pod \"auth-test-model-482dce-kserve-67d5796c5f-zpknh\" (UID: \"cb2a60ce-3718-4c59-8c72-7608c97b6027\") " pod="ai-tenant-e2e-shared-b-w3-482dce/auth-test-model-482dce-kserve-67d5796c5f-zpknh"
Sep 09 15:49:43.011323 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:43.011288    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb2a60ce-3718-4c59-8c72-7608c97b6027-tmp-dir\") pod \"auth-test-model-482dce-kserve-67d5796c5f-zpknh\" (UID: \"cb2a60ce-3718-4c59-8c72-7608c97b6027\") " pod="ai-tenant-e2e-shared-b-w3-482dce/auth-test-model-482dce-kserve-67d5796c5f-zpknh"
Sep 09 15:49:43.011323 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:43.011328    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb2a60ce-3718-4c59-8c72-7608c97b6027-tls-certs\") pod \"auth-test-model-482dce-kserve-67d5796c5f-zpknh\" (UID: \"cb2a60ce-3718-4c59-8c72-7608c97b6027\") " pod="ai-tenant-e2e-shared-b-w3-482dce/auth-test-model-482dce-kserve-67d5796c5f-zpknh"
Sep 09 15:49:43.011558 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:43.011354    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sb6r\" (UniqueName: \"kubernetes.io/projected/cb2a60ce-3718-4c59-8c72-7608c97b6027-kube-api-access-2sb6r\") pod \"auth-test-model-482dce-kserve-67d5796c5f-zpknh\" (UID: \"cb2a60ce-3718-4c59-8c72-7608c97b6027\") " pod="ai-tenant-e2e-shared-b-w3-482dce/auth-test-model-482dce-kserve-67d5796c5f-zpknh"
Sep 09 15:49:43.011558 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:43.011440    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb2a60ce-3718-4c59-8c72-7608c97b6027-model-cache\") pod \"auth-test-model-482dce-kserve-67d5796c5f-zpknh\" (UID: \"cb2a60ce-3718-4c59-8c72-7608c97b6027\") " pod="ai-tenant-e2e-shared-b-w3-482dce/auth-test-model-482dce-kserve-67d5796c5f-zpknh"
Sep 09 15:49:43.011558 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:43.011505    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb2a60ce-3718-4c59-8c72-7608c97b6027-home\") pod \"auth-test-model-482dce-kserve-67d5796c5f-zpknh\" (UID: \"cb2a60ce-3718-4c59-8c72-7608c97b6027\") " pod="ai-tenant-e2e-shared-b-w3-482dce/auth-test-model-482dce-kserve-67d5796c5f-zpknh"
Sep 09 15:49:43.011558 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:43.011531    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb2a60ce-3718-4c59-8c72-7608c97b6027-dshm\") pod \"auth-test-model-482dce-kserve-67d5796c5f-zpknh\" (UID: \"cb2a60ce-3718-4c59-8c72-7608c97b6027\") " pod="ai-tenant-e2e-shared-b-w3-482dce/auth-test-model-482dce-kserve-67d5796c5f-zpknh"
Sep 09 15:49:43.011822 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:43.011794    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb2a60ce-3718-4c59-8c72-7608c97b6027-tmp-dir\") pod \"auth-test-model-482dce-kserve-67d5796c5f-zpknh\" (UID: \"cb2a60ce-3718-4c59-8c72-7608c97b6027\") " pod="ai-tenant-e2e-shared-b-w3-482dce/auth-test-model-482dce-kserve-67d5796c5f-zpknh"
Sep 09 15:49:43.011898 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:43.011874    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb2a60ce-3718-4c59-8c72-7608c97b6027-model-cache\") pod \"auth-test-model-482dce-kserve-67d5796c5f-zpknh\" (UID: \"cb2a60ce-3718-4c59-8c72-7608c97b6027\") " pod="ai-tenant-e2e-shared-b-w3-482dce/auth-test-model-482dce-kserve-67d5796c5f-zpknh"
Sep 09 15:49:43.011946 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:43.011905    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb2a60ce-3718-4c59-8c72-7608c97b6027-home\") pod \"auth-test-model-482dce-kserve-67d5796c5f-zpknh\" (UID: \"cb2a60ce-3718-4c59-8c72-7608c97b6027\") " pod="ai-tenant-e2e-shared-b-w3-482dce/auth-test-model-482dce-kserve-67d5796c5f-zpknh"
Sep 09 15:49:43.013696 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:43.013668    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb2a60ce-3718-4c59-8c72-7608c97b6027-dshm\") pod \"auth-test-model-482dce-kserve-67d5796c5f-zpknh\" (UID: \"cb2a60ce-3718-4c59-8c72-7608c97b6027\") " pod="ai-tenant-e2e-shared-b-w3-482dce/auth-test-model-482dce-kserve-67d5796c5f-zpknh"
Sep 09 15:49:43.014114 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:43.014096    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb2a60ce-3718-4c59-8c72-7608c97b6027-tls-certs\") pod \"auth-test-model-482dce-kserve-67d5796c5f-zpknh\" (UID: \"cb2a60ce-3718-4c59-8c72-7608c97b6027\") " pod="ai-tenant-e2e-shared-b-w3-482dce/auth-test-model-482dce-kserve-67d5796c5f-zpknh"
Sep 09 15:49:43.024552 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:43.024527    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sb6r\" (UniqueName: \"kubernetes.io/projected/cb2a60ce-3718-4c59-8c72-7608c97b6027-kube-api-access-2sb6r\") pod \"auth-test-model-482dce-kserve-67d5796c5f-zpknh\" (UID: \"cb2a60ce-3718-4c59-8c72-7608c97b6027\") " pod="ai-tenant-e2e-shared-b-w3-482dce/auth-test-model-482dce-kserve-67d5796c5f-zpknh"
Sep 09 15:49:43.170593 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:43.170558    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-482dce/auth-test-model-482dce-kserve-67d5796c5f-zpknh"
Sep 09 15:49:43.447316 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:43.447279    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-482dce/auth-test-model-482dce-kserve-67d5796c5f-zpknh"]
Sep 09 15:49:43.448301 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:49:43.448274    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb2a60ce_3718_4c59_8c72_7608c97b6027.slice/crio-dc61e30a7803e5f79481c6d3700aa94191e36c5c37e3a64d48468da86b2065b7 WatchSource:0}: Error finding container dc61e30a7803e5f79481c6d3700aa94191e36c5c37e3a64d48468da86b2065b7: Status 404 returned error can't find the container with id dc61e30a7803e5f79481c6d3700aa94191e36c5c37e3a64d48468da86b2065b7
Sep 09 15:49:43.972173 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:43.972119    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-482dce/auth-test-model-482dce-kserve-67d5796c5f-zpknh" event={"ID":"cb2a60ce-3718-4c59-8c72-7608c97b6027","Type":"ContainerStarted","Data":"14cfe5f010c9398abbcb4404dac529a86744e519639aae22a75d4b65235601f0"}
Sep 09 15:49:43.972173 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:43.972175    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-482dce/auth-test-model-482dce-kserve-67d5796c5f-zpknh" event={"ID":"cb2a60ce-3718-4c59-8c72-7608c97b6027","Type":"ContainerStarted","Data":"dc61e30a7803e5f79481c6d3700aa94191e36c5c37e3a64d48468da86b2065b7"}
Sep 09 15:49:43.989853 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:43.989785    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-482dce/auth-test-model-482dce-kserve-67d5796c5f-zpknh" podStartSLOduration=1.9897666520000001 podStartE2EDuration="1.989766652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:49:43.987796287 +0000 UTC m=+1749.586690932" watchObservedRunningTime="2026-09-09 15:49:43.989766652 +0000 UTC m=+1749.588661295"
Sep 09 15:49:53.170815 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:53.170772    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-482dce/auth-test-model-482dce-kserve-67d5796c5f-zpknh"
Sep 09 15:49:53.171226 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:53.170832    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-482dce/auth-test-model-482dce-kserve-67d5796c5f-zpknh"
Sep 09 15:49:53.183692 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:53.183662    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-482dce/auth-test-model-482dce-kserve-67d5796c5f-zpknh"
Sep 09 15:49:54.043751 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:54.043701    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-482dce/auth-test-model-482dce-kserve-67d5796c5f-zpknh"
Sep 09 15:49:54.367434 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:54.367336    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-df30be-qdsh9"]
Sep 09 15:49:54.371000 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:54.370983    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-df30be-qdsh9"
Sep 09 15:49:54.373681 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:54.373660    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2zlsf\""
Sep 09 15:49:54.373831 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:54.373667    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 15:49:54.378266 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:54.378231    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-df30be-qdsh9"]
Sep 09 15:49:54.532631 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:54.532596    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8e3467f-2c61-4583-99b9-b06b99add19a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-df30be-qdsh9\" (UID: \"e8e3467f-2c61-4583-99b9-b06b99add19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-df30be-qdsh9"
Sep 09 15:49:54.633546 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:54.633456    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8e3467f-2c61-4583-99b9-b06b99add19a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-df30be-qdsh9\" (UID: \"e8e3467f-2c61-4583-99b9-b06b99add19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-df30be-qdsh9"
Sep 09 15:49:54.634165 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:54.634142    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8e3467f-2c61-4583-99b9-b06b99add19a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-df30be-qdsh9\" (UID: \"e8e3467f-2c61-4583-99b9-b06b99add19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-df30be-qdsh9"
Sep 09 15:49:54.683690 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:54.683656    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-df30be-qdsh9"
Sep 09 15:49:54.959617 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:49:54.959587    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e3467f_2c61_4583_99b9_b06b99add19a.slice/crio-5eb96d6d58ddfcd47eab415e69db7512c20c52f518a8a5612f25c46fe193ba76 WatchSource:0}: Error finding container 5eb96d6d58ddfcd47eab415e69db7512c20c52f518a8a5612f25c46fe193ba76: Status 404 returned error can't find the container with id 5eb96d6d58ddfcd47eab415e69db7512c20c52f518a8a5612f25c46fe193ba76
Sep 09 15:49:54.964460 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:54.964438    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-df30be-qdsh9"]
Sep 09 15:49:55.040053 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:55.040010    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-df30be-qdsh9" event={"ID":"e8e3467f-2c61-4583-99b9-b06b99add19a","Type":"ContainerStarted","Data":"5eb96d6d58ddfcd47eab415e69db7512c20c52f518a8a5612f25c46fe193ba76"}
Sep 09 15:49:56.047693 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:56.047661    2553 generic.go:356] "Generic (PLEG): container finished" podID="e8e3467f-2c61-4583-99b9-b06b99add19a" containerID="9c943d1f1b5756ef18e58f2f738a598f50f926cb8f761945aaa1a242a39c5c2c" exitCode=0
Sep 09 15:49:56.048098 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:56.047756    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-df30be-qdsh9" event={"ID":"e8e3467f-2c61-4583-99b9-b06b99add19a","Type":"ContainerDied","Data":"9c943d1f1b5756ef18e58f2f738a598f50f926cb8f761945aaa1a242a39c5c2c"}
Sep 09 15:49:57.195500 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:57.195462    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-df30be-qdsh9"
Sep 09 15:49:57.362437 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:57.362343    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8e3467f-2c61-4583-99b9-b06b99add19a-maas-api-service-ca\") pod \"e8e3467f-2c61-4583-99b9-b06b99add19a\" (UID: \"e8e3467f-2c61-4583-99b9-b06b99add19a\") "
Sep 09 15:49:57.362723 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:57.362698    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8e3467f-2c61-4583-99b9-b06b99add19a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e8e3467f-2c61-4583-99b9-b06b99add19a" (UID: "e8e3467f-2c61-4583-99b9-b06b99add19a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:49:57.463469 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:57.463430    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8e3467f-2c61-4583-99b9-b06b99add19a-maas-api-service-ca\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:49:58.061213 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:58.061178    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-df30be-qdsh9"
Sep 09 15:49:58.061406 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:58.061218    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-df30be-qdsh9" event={"ID":"e8e3467f-2c61-4583-99b9-b06b99add19a","Type":"ContainerDied","Data":"5eb96d6d58ddfcd47eab415e69db7512c20c52f518a8a5612f25c46fe193ba76"}
Sep 09 15:49:58.061406 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:49:58.061276    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5eb96d6d58ddfcd47eab415e69db7512c20c52f518a8a5612f25c46fe193ba76"
Sep 09 15:50:10.821230 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:10.821186    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-df30be-57ff546d6b-ts9mj"]
Sep 09 15:50:10.822286 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:10.821477    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-df30be-57ff546d6b-ts9mj" podUID="19eb8f8e-5347-4a51-8b90-cdbf2b1102f9" containerName="payload-pre-processing" containerID="cri-o://7e2d6ee1eea4877a6c786d237fc6c515b9cbaaa67fa58ddaf6f7ddb06a8508fa" gracePeriod=30
Sep 09 15:50:11.058925 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:11.058889    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-df30be-qdsh9"]
Sep 09 15:50:11.063619 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:11.063590    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-df30be-qdsh9"]
Sep 09 15:50:11.107604 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:11.107578    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-df30be-57ff546d6b-ts9mj"
Sep 09 15:50:11.141012 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:11.140972    2553 generic.go:356] "Generic (PLEG): container finished" podID="19eb8f8e-5347-4a51-8b90-cdbf2b1102f9" containerID="7e2d6ee1eea4877a6c786d237fc6c515b9cbaaa67fa58ddaf6f7ddb06a8508fa" exitCode=0
Sep 09 15:50:11.141215 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:11.141021    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-df30be-57ff546d6b-ts9mj"
Sep 09 15:50:11.141215 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:11.141023    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-df30be-57ff546d6b-ts9mj" event={"ID":"19eb8f8e-5347-4a51-8b90-cdbf2b1102f9","Type":"ContainerDied","Data":"7e2d6ee1eea4877a6c786d237fc6c515b9cbaaa67fa58ddaf6f7ddb06a8508fa"}
Sep 09 15:50:11.141215 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:11.141109    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-df30be-57ff546d6b-ts9mj" event={"ID":"19eb8f8e-5347-4a51-8b90-cdbf2b1102f9","Type":"ContainerDied","Data":"68ec637209149ea8457fefc94377ec7df44b5809a3d8beb04be63881dccfc64a"}
Sep 09 15:50:11.141215 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:11.141150    2553 scope.go:117] "RemoveContainer" containerID="7e2d6ee1eea4877a6c786d237fc6c515b9cbaaa67fa58ddaf6f7ddb06a8508fa"
Sep 09 15:50:11.152806 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:11.152784    2553 scope.go:117] "RemoveContainer" containerID="7e2d6ee1eea4877a6c786d237fc6c515b9cbaaa67fa58ddaf6f7ddb06a8508fa"
Sep 09 15:50:11.153171 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:50:11.153142    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e2d6ee1eea4877a6c786d237fc6c515b9cbaaa67fa58ddaf6f7ddb06a8508fa\": container with ID starting with 7e2d6ee1eea4877a6c786d237fc6c515b9cbaaa67fa58ddaf6f7ddb06a8508fa not found: ID does not exist" containerID="7e2d6ee1eea4877a6c786d237fc6c515b9cbaaa67fa58ddaf6f7ddb06a8508fa"
Sep 09 15:50:11.153310 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:11.153176    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e2d6ee1eea4877a6c786d237fc6c515b9cbaaa67fa58ddaf6f7ddb06a8508fa"} err="failed to get container status \"7e2d6ee1eea4877a6c786d237fc6c515b9cbaaa67fa58ddaf6f7ddb06a8508fa\": rpc error: code = NotFound desc = could not find container \"7e2d6ee1eea4877a6c786d237fc6c515b9cbaaa67fa58ddaf6f7ddb06a8508fa\": container with ID starting with 7e2d6ee1eea4877a6c786d237fc6c515b9cbaaa67fa58ddaf6f7ddb06a8508fa not found: ID does not exist"
Sep 09 15:50:11.199861 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:11.199825    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxfkl\" (UniqueName: \"kubernetes.io/projected/19eb8f8e-5347-4a51-8b90-cdbf2b1102f9-kube-api-access-qxfkl\") pod \"19eb8f8e-5347-4a51-8b90-cdbf2b1102f9\" (UID: \"19eb8f8e-5347-4a51-8b90-cdbf2b1102f9\") "
Sep 09 15:50:11.200046 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:11.199888    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19eb8f8e-5347-4a51-8b90-cdbf2b1102f9-plugins-config-volume\") pod \"19eb8f8e-5347-4a51-8b90-cdbf2b1102f9\" (UID: \"19eb8f8e-5347-4a51-8b90-cdbf2b1102f9\") "
Sep 09 15:50:11.200334 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:11.200305    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19eb8f8e-5347-4a51-8b90-cdbf2b1102f9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "19eb8f8e-5347-4a51-8b90-cdbf2b1102f9" (UID: "19eb8f8e-5347-4a51-8b90-cdbf2b1102f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:50:11.202052 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:11.202027    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19eb8f8e-5347-4a51-8b90-cdbf2b1102f9-kube-api-access-qxfkl" (OuterVolumeSpecName: "kube-api-access-qxfkl") pod "19eb8f8e-5347-4a51-8b90-cdbf2b1102f9" (UID: "19eb8f8e-5347-4a51-8b90-cdbf2b1102f9"). InnerVolumeSpecName "kube-api-access-qxfkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:50:11.300925 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:11.300874    2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19eb8f8e-5347-4a51-8b90-cdbf2b1102f9-plugins-config-volume\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:50:11.300925 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:11.300915    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxfkl\" (UniqueName: \"kubernetes.io/projected/19eb8f8e-5347-4a51-8b90-cdbf2b1102f9-kube-api-access-qxfkl\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:50:11.468327 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:11.468285    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-df30be-57ff546d6b-ts9mj"]
Sep 09 15:50:11.473046 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:11.473015    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-df30be-57ff546d6b-ts9mj"]
Sep 09 15:50:12.963306 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:12.963269    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19eb8f8e-5347-4a51-8b90-cdbf2b1102f9" path="/var/lib/kubelet/pods/19eb8f8e-5347-4a51-8b90-cdbf2b1102f9/volumes"
Sep 09 15:50:12.963846 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:12.963824    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8e3467f-2c61-4583-99b9-b06b99add19a" path="/var/lib/kubelet/pods/e8e3467f-2c61-4583-99b9-b06b99add19a/volumes"
Sep 09 15:50:18.125754 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:18.125706    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"]
Sep 09 15:50:18.126233 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:18.125928    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k" podUID="3b0eec3f-27bc-4191-a148-4c5a8a531611" containerName="istio-proxy" containerID="cri-o://5176c530d58c9649dba3d393258d041bfb3c6d023cdf557f42e60bd47dc2427d" gracePeriod=30
Sep 09 15:50:23.380317 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:23.380283    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"
Sep 09 15:50:23.524255 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:23.524208    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0eec3f-27bc-4191-a148-4c5a8a531611-workload-certs\") pod \"3b0eec3f-27bc-4191-a148-4c5a8a531611\" (UID: \"3b0eec3f-27bc-4191-a148-4c5a8a531611\") "
Sep 09 15:50:23.524255 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:23.524271    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0eec3f-27bc-4191-a148-4c5a8a531611-workload-socket\") pod \"3b0eec3f-27bc-4191-a148-4c5a8a531611\" (UID: \"3b0eec3f-27bc-4191-a148-4c5a8a531611\") "
Sep 09 15:50:23.524541 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:23.524319    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0eec3f-27bc-4191-a148-4c5a8a531611-istio-data\") pod \"3b0eec3f-27bc-4191-a148-4c5a8a531611\" (UID: \"3b0eec3f-27bc-4191-a148-4c5a8a531611\") "
Sep 09 15:50:23.524541 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:23.524344    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0eec3f-27bc-4191-a148-4c5a8a531611-istiod-ca-cert\") pod \"3b0eec3f-27bc-4191-a148-4c5a8a531611\" (UID: \"3b0eec3f-27bc-4191-a148-4c5a8a531611\") "
Sep 09 15:50:23.524541 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:23.524395    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0eec3f-27bc-4191-a148-4c5a8a531611-credential-socket\") pod \"3b0eec3f-27bc-4191-a148-4c5a8a531611\" (UID: \"3b0eec3f-27bc-4191-a148-4c5a8a531611\") "
Sep 09 15:50:23.524541 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:23.524459    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b0eec3f-27bc-4191-a148-4c5a8a531611-istio-token\") pod \"3b0eec3f-27bc-4191-a148-4c5a8a531611\" (UID: \"3b0eec3f-27bc-4191-a148-4c5a8a531611\") "
Sep 09 15:50:23.524541 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:23.524500    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0eec3f-27bc-4191-a148-4c5a8a531611-istio-podinfo\") pod \"3b0eec3f-27bc-4191-a148-4c5a8a531611\" (UID: \"3b0eec3f-27bc-4191-a148-4c5a8a531611\") "
Sep 09 15:50:23.524833 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:23.524523    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0eec3f-27bc-4191-a148-4c5a8a531611-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3b0eec3f-27bc-4191-a148-4c5a8a531611" (UID: "3b0eec3f-27bc-4191-a148-4c5a8a531611"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:50:23.524833 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:23.524549    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p278h\" (UniqueName: \"kubernetes.io/projected/3b0eec3f-27bc-4191-a148-4c5a8a531611-kube-api-access-p278h\") pod \"3b0eec3f-27bc-4191-a148-4c5a8a531611\" (UID: \"3b0eec3f-27bc-4191-a148-4c5a8a531611\") "
Sep 09 15:50:23.524833 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:23.524637    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0eec3f-27bc-4191-a148-4c5a8a531611-istio-envoy\") pod \"3b0eec3f-27bc-4191-a148-4c5a8a531611\" (UID: \"3b0eec3f-27bc-4191-a148-4c5a8a531611\") "
Sep 09 15:50:23.524833 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:23.524668    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0eec3f-27bc-4191-a148-4c5a8a531611-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3b0eec3f-27bc-4191-a148-4c5a8a531611" (UID: "3b0eec3f-27bc-4191-a148-4c5a8a531611"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:50:23.524833 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:23.524703    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0eec3f-27bc-4191-a148-4c5a8a531611-istio-data" (OuterVolumeSpecName: "istio-data") pod "3b0eec3f-27bc-4191-a148-4c5a8a531611" (UID: "3b0eec3f-27bc-4191-a148-4c5a8a531611"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:50:23.524833 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:23.524775    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0eec3f-27bc-4191-a148-4c5a8a531611-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3b0eec3f-27bc-4191-a148-4c5a8a531611" (UID: "3b0eec3f-27bc-4191-a148-4c5a8a531611"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:50:23.524833 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:23.524796    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b0eec3f-27bc-4191-a148-4c5a8a531611-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3b0eec3f-27bc-4191-a148-4c5a8a531611" (UID: "3b0eec3f-27bc-4191-a148-4c5a8a531611"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:50:23.525144 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:23.524999    2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0eec3f-27bc-4191-a148-4c5a8a531611-workload-certs\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:50:23.525144 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:23.525012    2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0eec3f-27bc-4191-a148-4c5a8a531611-workload-socket\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:50:23.525144 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:23.525026    2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0eec3f-27bc-4191-a148-4c5a8a531611-istio-data\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:50:23.525144 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:23.525044    2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0eec3f-27bc-4191-a148-4c5a8a531611-istiod-ca-cert\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:50:23.525144 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:23.525059    2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0eec3f-27bc-4191-a148-4c5a8a531611-credential-socket\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:50:23.526770 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:23.526726    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b0eec3f-27bc-4191-a148-4c5a8a531611-istio-token" (OuterVolumeSpecName: "istio-token") pod "3b0eec3f-27bc-4191-a148-4c5a8a531611" (UID: "3b0eec3f-27bc-4191-a148-4c5a8a531611"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:50:23.526878 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:23.526857    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b0eec3f-27bc-4191-a148-4c5a8a531611-kube-api-access-p278h" (OuterVolumeSpecName: "kube-api-access-p278h") pod "3b0eec3f-27bc-4191-a148-4c5a8a531611" (UID: "3b0eec3f-27bc-4191-a148-4c5a8a531611"). InnerVolumeSpecName "kube-api-access-p278h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:50:23.527185 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:23.527163    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0eec3f-27bc-4191-a148-4c5a8a531611-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3b0eec3f-27bc-4191-a148-4c5a8a531611" (UID: "3b0eec3f-27bc-4191-a148-4c5a8a531611"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:50:23.527185 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:23.527162    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b0eec3f-27bc-4191-a148-4c5a8a531611-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3b0eec3f-27bc-4191-a148-4c5a8a531611" (UID: "3b0eec3f-27bc-4191-a148-4c5a8a531611"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 15:50:23.625608 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:23.625570    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p278h\" (UniqueName: \"kubernetes.io/projected/3b0eec3f-27bc-4191-a148-4c5a8a531611-kube-api-access-p278h\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:50:23.625608 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:23.625602    2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0eec3f-27bc-4191-a148-4c5a8a531611-istio-envoy\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:50:23.625608 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:23.625612    2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b0eec3f-27bc-4191-a148-4c5a8a531611-istio-token\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:50:23.625878 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:23.625620    2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0eec3f-27bc-4191-a148-4c5a8a531611-istio-podinfo\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:50:24.218943 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:24.218907    2553 generic.go:356] "Generic (PLEG): container finished" podID="3b0eec3f-27bc-4191-a148-4c5a8a531611" containerID="5176c530d58c9649dba3d393258d041bfb3c6d023cdf557f42e60bd47dc2427d" exitCode=0
Sep 09 15:50:24.219147 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:24.218963    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"
Sep 09 15:50:24.219147 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:24.218991    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k" event={"ID":"3b0eec3f-27bc-4191-a148-4c5a8a531611","Type":"ContainerDied","Data":"5176c530d58c9649dba3d393258d041bfb3c6d023cdf557f42e60bd47dc2427d"}
Sep 09 15:50:24.219147 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:24.219047    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k" event={"ID":"3b0eec3f-27bc-4191-a148-4c5a8a531611","Type":"ContainerDied","Data":"8b7d6776a47e7dfd87c1125c16fa23c167f9d95cd551bb5339312375d9c5726f"}
Sep 09 15:50:24.219147 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:24.219076    2553 scope.go:117] "RemoveContainer" containerID="5176c530d58c9649dba3d393258d041bfb3c6d023cdf557f42e60bd47dc2427d"
Sep 09 15:50:24.228506 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:24.228485    2553 scope.go:117] "RemoveContainer" containerID="635dd9ae07ff8cfe25f6de4ddff68a105eab182d691bd46247c98ed82d369912"
Sep 09 15:50:24.239971 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:24.239948    2553 scope.go:117] "RemoveContainer" containerID="5176c530d58c9649dba3d393258d041bfb3c6d023cdf557f42e60bd47dc2427d"
Sep 09 15:50:24.240389 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:50:24.240362    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5176c530d58c9649dba3d393258d041bfb3c6d023cdf557f42e60bd47dc2427d\": container with ID starting with 5176c530d58c9649dba3d393258d041bfb3c6d023cdf557f42e60bd47dc2427d not found: ID does not exist" containerID="5176c530d58c9649dba3d393258d041bfb3c6d023cdf557f42e60bd47dc2427d"
Sep 09 15:50:24.240479 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:24.240400    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5176c530d58c9649dba3d393258d041bfb3c6d023cdf557f42e60bd47dc2427d"} err="failed to get container status \"5176c530d58c9649dba3d393258d041bfb3c6d023cdf557f42e60bd47dc2427d\": rpc error: code = NotFound desc = could not find container \"5176c530d58c9649dba3d393258d041bfb3c6d023cdf557f42e60bd47dc2427d\": container with ID starting with 5176c530d58c9649dba3d393258d041bfb3c6d023cdf557f42e60bd47dc2427d not found: ID does not exist"
Sep 09 15:50:24.240479 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:24.240424    2553 scope.go:117] "RemoveContainer" containerID="635dd9ae07ff8cfe25f6de4ddff68a105eab182d691bd46247c98ed82d369912"
Sep 09 15:50:24.240755 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:50:24.240711    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"635dd9ae07ff8cfe25f6de4ddff68a105eab182d691bd46247c98ed82d369912\": container with ID starting with 635dd9ae07ff8cfe25f6de4ddff68a105eab182d691bd46247c98ed82d369912 not found: ID does not exist" containerID="635dd9ae07ff8cfe25f6de4ddff68a105eab182d691bd46247c98ed82d369912"
Sep 09 15:50:24.240832 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:24.240763    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"635dd9ae07ff8cfe25f6de4ddff68a105eab182d691bd46247c98ed82d369912"} err="failed to get container status \"635dd9ae07ff8cfe25f6de4ddff68a105eab182d691bd46247c98ed82d369912\": rpc error: code = NotFound desc = could not find container \"635dd9ae07ff8cfe25f6de4ddff68a105eab182d691bd46247c98ed82d369912\": container with ID starting with 635dd9ae07ff8cfe25f6de4ddff68a105eab182d691bd46247c98ed82d369912 not found: ID does not exist"
Sep 09 15:50:24.242813 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:24.242504    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"]
Sep 09 15:50:24.244908 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:24.244885    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-df30be-openshift-default-844c65f5d4-blm5k"]
Sep 09 15:50:24.962675 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:24.962639    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b0eec3f-27bc-4191-a148-4c5a8a531611" path="/var/lib/kubelet/pods/3b0eec3f-27bc-4191-a148-4c5a8a531611/volumes"
Sep 09 15:50:35.128065 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:35.128029    2553 scope.go:117] "RemoveContainer" containerID="be1ea39a3a4f07ee6caf1825e187ea1c31c3a1e87524f169209560418e930e5d"
Sep 09 15:50:35.136931 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:35.136898    2553 scope.go:117] "RemoveContainer" containerID="93e40d7a2f075fa65990dcacf4684e4a4c41f2b9689c33fd0e026a03e8122453"
Sep 09 15:50:37.654722 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.654685    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"]
Sep 09 15:50:37.655282 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.655261    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19eb8f8e-5347-4a51-8b90-cdbf2b1102f9" containerName="payload-pre-processing"
Sep 09 15:50:37.655367 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.655285    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="19eb8f8e-5347-4a51-8b90-cdbf2b1102f9" containerName="payload-pre-processing"
Sep 09 15:50:37.655367 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.655304    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b0eec3f-27bc-4191-a148-4c5a8a531611" containerName="istio-proxy"
Sep 09 15:50:37.655367 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.655313    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b0eec3f-27bc-4191-a148-4c5a8a531611" containerName="istio-proxy"
Sep 09 15:50:37.655367 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.655331    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8e3467f-2c61-4583-99b9-b06b99add19a" containerName="revoke-keys"
Sep 09 15:50:37.655367 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.655342    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8e3467f-2c61-4583-99b9-b06b99add19a" containerName="revoke-keys"
Sep 09 15:50:37.655367 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.655356    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b0eec3f-27bc-4191-a148-4c5a8a531611" containerName="istio-proxy"
Sep 09 15:50:37.655367 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.655365    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b0eec3f-27bc-4191-a148-4c5a8a531611" containerName="istio-proxy"
Sep 09 15:50:37.655702 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.655460    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8e3467f-2c61-4583-99b9-b06b99add19a" containerName="revoke-keys"
Sep 09 15:50:37.655702 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.655473    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b0eec3f-27bc-4191-a148-4c5a8a531611" containerName="istio-proxy"
Sep 09 15:50:37.655702 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.655481    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b0eec3f-27bc-4191-a148-4c5a8a531611" containerName="istio-proxy"
Sep 09 15:50:37.655702 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.655495    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="19eb8f8e-5347-4a51-8b90-cdbf2b1102f9" containerName="payload-pre-processing"
Sep 09 15:50:37.658551 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.658528    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"
Sep 09 15:50:37.661171 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.661144    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-ce9689-openshift-default-dockercfg-9sjbk\""
Sep 09 15:50:37.670049 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.670021    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"]
Sep 09 15:50:37.768750 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.768684    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-istio-token\") pod \"e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv\" (UID: \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\") " pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"
Sep 09 15:50:37.768927 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.768752    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n77tn\" (UniqueName: \"kubernetes.io/projected/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-kube-api-access-n77tn\") pod \"e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv\" (UID: \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\") " pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"
Sep 09 15:50:37.768927 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.768790    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-credential-socket\") pod \"e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv\" (UID: \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\") " pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"
Sep 09 15:50:37.768927 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.768812    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-istio-podinfo\") pod \"e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv\" (UID: \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\") " pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"
Sep 09 15:50:37.768927 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.768911    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-workload-certs\") pod \"e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv\" (UID: \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\") " pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"
Sep 09 15:50:37.769181 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.768966    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-workload-socket\") pod \"e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv\" (UID: \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\") " pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"
Sep 09 15:50:37.769181 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.768996    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-istiod-ca-cert\") pod \"e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv\" (UID: \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\") " pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"
Sep 09 15:50:37.769181 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.769060    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-istio-envoy\") pod \"e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv\" (UID: \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\") " pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"
Sep 09 15:50:37.769181 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.769159    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-istio-data\") pod \"e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv\" (UID: \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\") " pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"
Sep 09 15:50:37.869830 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.869793    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-istio-envoy\") pod \"e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv\" (UID: \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\") " pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"
Sep 09 15:50:37.870043 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.869891    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-istio-data\") pod \"e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv\" (UID: \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\") " pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"
Sep 09 15:50:37.870043 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.869920    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-istio-token\") pod \"e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv\" (UID: \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\") " pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"
Sep 09 15:50:37.870043 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.869939    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n77tn\" (UniqueName: \"kubernetes.io/projected/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-kube-api-access-n77tn\") pod \"e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv\" (UID: \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\") " pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"
Sep 09 15:50:37.870043 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.869956    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-credential-socket\") pod \"e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv\" (UID: \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\") " pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"
Sep 09 15:50:37.870043 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.869971    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-istio-podinfo\") pod \"e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv\" (UID: \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\") " pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"
Sep 09 15:50:37.870043 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.870027    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-workload-certs\") pod \"e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv\" (UID: \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\") " pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"
Sep 09 15:50:37.870403 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.870070    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-workload-socket\") pod \"e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv\" (UID: \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\") " pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"
Sep 09 15:50:37.870403 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.870094    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-istiod-ca-cert\") pod \"e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv\" (UID: \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\") " pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"
Sep 09 15:50:37.870513 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.870398    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-credential-socket\") pod \"e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv\" (UID: \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\") " pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"
Sep 09 15:50:37.870634 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.870604    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-workload-certs\") pod \"e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv\" (UID: \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\") " pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"
Sep 09 15:50:37.870776 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.870653    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-istio-data\") pod \"e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv\" (UID: \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\") " pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"
Sep 09 15:50:37.870776 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.870767    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-workload-socket\") pod \"e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv\" (UID: \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\") " pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"
Sep 09 15:50:37.870914 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.870817    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-istiod-ca-cert\") pod \"e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv\" (UID: \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\") " pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"
Sep 09 15:50:37.872859 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.872840    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-istio-envoy\") pod \"e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv\" (UID: \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\") " pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"
Sep 09 15:50:37.873001 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.872982    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-istio-podinfo\") pod \"e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv\" (UID: \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\") " pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"
Sep 09 15:50:37.877358 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.877335    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-istio-token\") pod \"e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv\" (UID: \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\") " pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"
Sep 09 15:50:37.877460 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.877415    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n77tn\" (UniqueName: \"kubernetes.io/projected/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-kube-api-access-n77tn\") pod \"e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv\" (UID: \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\") " pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"
Sep 09 15:50:37.974461 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:37.974424    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"
Sep 09 15:50:38.251960 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:38.251898    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"]
Sep 09 15:50:38.254003 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:50:38.253959    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79946caa_e46d_4320_a5ec_80ad8a2c7bd9.slice/crio-8007e64d9a45b69be930ba9e33912a27ff57bc059200a8dd3dd5d7b05d3f3f28 WatchSource:0}: Error finding container 8007e64d9a45b69be930ba9e33912a27ff57bc059200a8dd3dd5d7b05d3f3f28: Status 404 returned error can't find the container with id 8007e64d9a45b69be930ba9e33912a27ff57bc059200a8dd3dd5d7b05d3f3f28
Sep 09 15:50:38.256673 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:38.256633    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:50:38.256771 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:38.256718    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:50:38.256824 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:38.256781    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:50:38.311809 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:38.311786    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv" event={"ID":"79946caa-e46d-4320-a5ec-80ad8a2c7bd9","Type":"ContainerStarted","Data":"8007e64d9a45b69be930ba9e33912a27ff57bc059200a8dd3dd5d7b05d3f3f28"}
Sep 09 15:50:39.318537 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:39.318496    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv" event={"ID":"79946caa-e46d-4320-a5ec-80ad8a2c7bd9","Type":"ContainerStarted","Data":"a65ce739ef66a78f818233f78d1198d2fc6301ef17918dd6eea9eb9869820593"}
Sep 09 15:50:39.340575 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:39.340510    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv" podStartSLOduration=2.340492838 podStartE2EDuration="2.340492838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:50:39.338337605 +0000 UTC m=+1804.937232250" watchObservedRunningTime="2026-09-09 15:50:39.340492838 +0000 UTC m=+1804.939387482"
Sep 09 15:50:39.975425 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:39.975391    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"
Sep 09 15:50:39.980706 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:39.980680    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"
Sep 09 15:50:40.325279 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:40.325196    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"
Sep 09 15:50:40.326278 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:40.326258    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"
Sep 09 15:50:43.580029 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:43.579928    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ce9689-9ff9b6fb5-ptgdt"]
Sep 09 15:50:43.584325 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:43.584291    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ce9689-9ff9b6fb5-ptgdt"
Sep 09 15:50:43.587189 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:43.587164    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-ce9689\""
Sep 09 15:50:43.587387 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:43.587208    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-ce9689-dockercfg-dvp9c\""
Sep 09 15:50:43.596702 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:43.596676    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ce9689-9ff9b6fb5-ptgdt"]
Sep 09 15:50:43.629884 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:43.629845    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ecd926-8585-4bd7-824d-c625cb47671c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ce9689-9ff9b6fb5-ptgdt\" (UID: \"67ecd926-8585-4bd7-824d-c625cb47671c\") " pod="openshift-ingress/payload-processing-e2e-api-b-ce9689-9ff9b6fb5-ptgdt"
Sep 09 15:50:43.630097 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:43.629983    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x55vq\" (UniqueName: \"kubernetes.io/projected/67ecd926-8585-4bd7-824d-c625cb47671c-kube-api-access-x55vq\") pod \"payload-processing-e2e-api-b-ce9689-9ff9b6fb5-ptgdt\" (UID: \"67ecd926-8585-4bd7-824d-c625cb47671c\") " pod="openshift-ingress/payload-processing-e2e-api-b-ce9689-9ff9b6fb5-ptgdt"
Sep 09 15:50:43.730664 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:43.730628    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ecd926-8585-4bd7-824d-c625cb47671c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ce9689-9ff9b6fb5-ptgdt\" (UID: \"67ecd926-8585-4bd7-824d-c625cb47671c\") " pod="openshift-ingress/payload-processing-e2e-api-b-ce9689-9ff9b6fb5-ptgdt"
Sep 09 15:50:43.730910 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:43.730716    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x55vq\" (UniqueName: \"kubernetes.io/projected/67ecd926-8585-4bd7-824d-c625cb47671c-kube-api-access-x55vq\") pod \"payload-processing-e2e-api-b-ce9689-9ff9b6fb5-ptgdt\" (UID: \"67ecd926-8585-4bd7-824d-c625cb47671c\") " pod="openshift-ingress/payload-processing-e2e-api-b-ce9689-9ff9b6fb5-ptgdt"
Sep 09 15:50:43.731373 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:43.731351    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ecd926-8585-4bd7-824d-c625cb47671c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ce9689-9ff9b6fb5-ptgdt\" (UID: \"67ecd926-8585-4bd7-824d-c625cb47671c\") " pod="openshift-ingress/payload-processing-e2e-api-b-ce9689-9ff9b6fb5-ptgdt"
Sep 09 15:50:43.738863 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:43.738828    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x55vq\" (UniqueName: \"kubernetes.io/projected/67ecd926-8585-4bd7-824d-c625cb47671c-kube-api-access-x55vq\") pod \"payload-processing-e2e-api-b-ce9689-9ff9b6fb5-ptgdt\" (UID: \"67ecd926-8585-4bd7-824d-c625cb47671c\") " pod="openshift-ingress/payload-processing-e2e-api-b-ce9689-9ff9b6fb5-ptgdt"
Sep 09 15:50:43.902185 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:43.902086    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ce9689-9ff9b6fb5-ptgdt"
Sep 09 15:50:44.175187 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:44.175158    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ce9689-9ff9b6fb5-ptgdt"]
Sep 09 15:50:44.177114 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:50:44.177085    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67ecd926_8585_4bd7_824d_c625cb47671c.slice/crio-5f0d84ae724e9fcee3c4f25d908044dfc60ad9365b3a2e9e324e77847a84bf6e WatchSource:0}: Error finding container 5f0d84ae724e9fcee3c4f25d908044dfc60ad9365b3a2e9e324e77847a84bf6e: Status 404 returned error can't find the container with id 5f0d84ae724e9fcee3c4f25d908044dfc60ad9365b3a2e9e324e77847a84bf6e
Sep 09 15:50:44.351498 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:44.351457    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ce9689-9ff9b6fb5-ptgdt" event={"ID":"67ecd926-8585-4bd7-824d-c625cb47671c","Type":"ContainerStarted","Data":"ca9f12a0ff44f8d5850eb5d9abca2e6fde116b4ede6f4332e553600af629c42d"}
Sep 09 15:50:44.351684 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:44.351537    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-ce9689-9ff9b6fb5-ptgdt"
Sep 09 15:50:44.351684 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:44.351558    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ce9689-9ff9b6fb5-ptgdt" event={"ID":"67ecd926-8585-4bd7-824d-c625cb47671c","Type":"ContainerStarted","Data":"5f0d84ae724e9fcee3c4f25d908044dfc60ad9365b3a2e9e324e77847a84bf6e"}
Sep 09 15:50:44.368044 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:44.367990    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-ce9689-9ff9b6fb5-ptgdt" podStartSLOduration=1.367974558 podStartE2EDuration="1.367974558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:50:44.36584238 +0000 UTC m=+1809.964737023" watchObservedRunningTime="2026-09-09 15:50:44.367974558 +0000 UTC m=+1809.966869201"
Sep 09 15:50:55.360061 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:50:55.360021    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-ce9689-9ff9b6fb5-ptgdt"
Sep 09 15:51:03.459562 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:03.459527    2553 generic.go:356] "Generic (PLEG): container finished" podID="79946caa-e46d-4320-a5ec-80ad8a2c7bd9" containerID="a65ce739ef66a78f818233f78d1198d2fc6301ef17918dd6eea9eb9869820593" exitCode=0
Sep 09 15:51:03.460026 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:03.459605    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv" event={"ID":"79946caa-e46d-4320-a5ec-80ad8a2c7bd9","Type":"ContainerDied","Data":"a65ce739ef66a78f818233f78d1198d2fc6301ef17918dd6eea9eb9869820593"}
Sep 09 15:51:03.460026 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:03.460023    2553 scope.go:117] "RemoveContainer" containerID="a65ce739ef66a78f818233f78d1198d2fc6301ef17918dd6eea9eb9869820593"
Sep 09 15:51:03.461244 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:03.461217    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:51:03.461315 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:03.461290    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:51:03.461352 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:03.461322    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:51:04.478457 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:04.478305    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv" event={"ID":"79946caa-e46d-4320-a5ec-80ad8a2c7bd9","Type":"ContainerStarted","Data":"40beab8a1b12c716de7200bdf99b5a7a6f220878b0b6817d4a14fcfc4b472555"}
Sep 09 15:51:04.975047 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:04.975009    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"
Sep 09 15:51:04.975235 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:04.975191    2553 patch_prober.go:28] interesting pod/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 09 15:51:04.975285 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:04.975252    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv" podUID="79946caa-e46d-4320-a5ec-80ad8a2c7bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 09 15:51:05.975489 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:05.975444    2553 patch_prober.go:28] interesting pod/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 09 15:51:05.975938 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:05.975552    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv" podUID="79946caa-e46d-4320-a5ec-80ad8a2c7bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 09 15:51:06.975296 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:06.975258    2553 patch_prober.go:28] interesting pod/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 09 15:51:06.975480 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:06.975315    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv" podUID="79946caa-e46d-4320-a5ec-80ad8a2c7bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 09 15:51:07.975581 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:07.975531    2553 patch_prober.go:28] interesting pod/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 09 15:51:07.976101 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:07.975603    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv" podUID="79946caa-e46d-4320-a5ec-80ad8a2c7bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 09 15:51:08.975888 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:08.975851    2553 patch_prober.go:28] interesting pod/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 09 15:51:08.976276 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:08.975928    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv" podUID="79946caa-e46d-4320-a5ec-80ad8a2c7bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 09 15:51:09.979481 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:09.979448    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"
Sep 09 15:51:09.979880 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:09.979727    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"
Sep 09 15:51:09.980404 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:09.980385    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"
Sep 09 15:51:10.717256 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:10.717218    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ac2fa-r7wbq"]
Sep 09 15:51:10.720912 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:10.720893    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ac2fa-r7wbq"
Sep 09 15:51:10.723492 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:10.723473    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 15:51:10.723492 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:10.723486    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2zlsf\""
Sep 09 15:51:10.727800 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:10.727712    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ac2fa-r7wbq"]
Sep 09 15:51:10.833201 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:10.833150    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/043fa9e6-b1ae-446b-8099-63f44ac57d9e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3ac2fa-r7wbq\" (UID: \"043fa9e6-b1ae-446b-8099-63f44ac57d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ac2fa-r7wbq"
Sep 09 15:51:10.934654 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:10.934610    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/043fa9e6-b1ae-446b-8099-63f44ac57d9e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3ac2fa-r7wbq\" (UID: \"043fa9e6-b1ae-446b-8099-63f44ac57d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ac2fa-r7wbq"
Sep 09 15:51:10.935290 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:10.935270    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/043fa9e6-b1ae-446b-8099-63f44ac57d9e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3ac2fa-r7wbq\" (UID: \"043fa9e6-b1ae-446b-8099-63f44ac57d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ac2fa-r7wbq"
Sep 09 15:51:11.034511 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:11.034414    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ac2fa-r7wbq"
Sep 09 15:51:11.314076 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:11.314046    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ac2fa-r7wbq"]
Sep 09 15:51:11.316214 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:51:11.316189    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod043fa9e6_b1ae_446b_8099_63f44ac57d9e.slice/crio-891049e8dd512ce6a594ef41ac27aa06d3de4a8f76a2f7fd3278bec3838e087b WatchSource:0}: Error finding container 891049e8dd512ce6a594ef41ac27aa06d3de4a8f76a2f7fd3278bec3838e087b: Status 404 returned error can't find the container with id 891049e8dd512ce6a594ef41ac27aa06d3de4a8f76a2f7fd3278bec3838e087b
Sep 09 15:51:11.523474 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:11.523434    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ac2fa-r7wbq" event={"ID":"043fa9e6-b1ae-446b-8099-63f44ac57d9e","Type":"ContainerStarted","Data":"891049e8dd512ce6a594ef41ac27aa06d3de4a8f76a2f7fd3278bec3838e087b"}
Sep 09 15:51:12.530546 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:12.530510    2553 generic.go:356] "Generic (PLEG): container finished" podID="043fa9e6-b1ae-446b-8099-63f44ac57d9e" containerID="8da24a2f0e920a067565a58c5e0a83da6c0105a5adb14b68214f6ac28005dc9b" exitCode=0
Sep 09 15:51:12.530953 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:12.530594    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ac2fa-r7wbq" event={"ID":"043fa9e6-b1ae-446b-8099-63f44ac57d9e","Type":"ContainerDied","Data":"8da24a2f0e920a067565a58c5e0a83da6c0105a5adb14b68214f6ac28005dc9b"}
Sep 09 15:51:13.676931 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:13.676897    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ac2fa-r7wbq"
Sep 09 15:51:13.766310 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:13.766274    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/043fa9e6-b1ae-446b-8099-63f44ac57d9e-maas-api-service-ca\") pod \"043fa9e6-b1ae-446b-8099-63f44ac57d9e\" (UID: \"043fa9e6-b1ae-446b-8099-63f44ac57d9e\") "
Sep 09 15:51:13.766657 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:13.766633    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/043fa9e6-b1ae-446b-8099-63f44ac57d9e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "043fa9e6-b1ae-446b-8099-63f44ac57d9e" (UID: "043fa9e6-b1ae-446b-8099-63f44ac57d9e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:51:13.867507 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:13.867417    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/043fa9e6-b1ae-446b-8099-63f44ac57d9e-maas-api-service-ca\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:51:14.543440 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:14.543409    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ac2fa-r7wbq"
Sep 09 15:51:14.543653 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:14.543447    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ac2fa-r7wbq" event={"ID":"043fa9e6-b1ae-446b-8099-63f44ac57d9e","Type":"ContainerDied","Data":"891049e8dd512ce6a594ef41ac27aa06d3de4a8f76a2f7fd3278bec3838e087b"}
Sep 09 15:51:14.543653 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:14.543494    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="891049e8dd512ce6a594ef41ac27aa06d3de4a8f76a2f7fd3278bec3838e087b"
Sep 09 15:51:26.484690 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:26.484645    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ac2fa-r7wbq"]
Sep 09 15:51:26.487420 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:26.487396    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ac2fa-r7wbq"]
Sep 09 15:51:26.969013 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:26.968978    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="043fa9e6-b1ae-446b-8099-63f44ac57d9e" path="/var/lib/kubelet/pods/043fa9e6-b1ae-446b-8099-63f44ac57d9e/volumes"
Sep 09 15:51:33.284244 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:33.284210    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ce9689-dxhfz"]
Sep 09 15:51:33.284632 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:33.284586    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="043fa9e6-b1ae-446b-8099-63f44ac57d9e" containerName="revoke-keys"
Sep 09 15:51:33.284632 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:33.284597    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="043fa9e6-b1ae-446b-8099-63f44ac57d9e" containerName="revoke-keys"
Sep 09 15:51:33.284704 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:33.284672    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="043fa9e6-b1ae-446b-8099-63f44ac57d9e" containerName="revoke-keys"
Sep 09 15:51:33.288038 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:33.288020    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ce9689-dxhfz"
Sep 09 15:51:33.290557 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:33.290527    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 15:51:33.290672 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:33.290626    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2zlsf\""
Sep 09 15:51:33.295184 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:33.295157    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ce9689-dxhfz"]
Sep 09 15:51:33.472218 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:33.472177    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/932dcde3-bb83-44cb-8c51-7cfa1744b0dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ce9689-dxhfz\" (UID: \"932dcde3-bb83-44cb-8c51-7cfa1744b0dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ce9689-dxhfz"
Sep 09 15:51:33.573414 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:33.573371    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/932dcde3-bb83-44cb-8c51-7cfa1744b0dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ce9689-dxhfz\" (UID: \"932dcde3-bb83-44cb-8c51-7cfa1744b0dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ce9689-dxhfz"
Sep 09 15:51:33.574094 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:33.574072    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/932dcde3-bb83-44cb-8c51-7cfa1744b0dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ce9689-dxhfz\" (UID: \"932dcde3-bb83-44cb-8c51-7cfa1744b0dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ce9689-dxhfz"
Sep 09 15:51:33.601212 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:33.601174    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ce9689-dxhfz"
Sep 09 15:51:33.875543 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:33.875517    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ce9689-dxhfz"]
Sep 09 15:51:34.663632 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:34.663599    2553 generic.go:356] "Generic (PLEG): container finished" podID="932dcde3-bb83-44cb-8c51-7cfa1744b0dc" containerID="1c609b61279a61678b642042b71f7b6a57aafd74b61db47c1712900b780ffaaf" exitCode=0
Sep 09 15:51:34.664047 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:34.663687    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ce9689-dxhfz" event={"ID":"932dcde3-bb83-44cb-8c51-7cfa1744b0dc","Type":"ContainerDied","Data":"1c609b61279a61678b642042b71f7b6a57aafd74b61db47c1712900b780ffaaf"}
Sep 09 15:51:34.664047 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:34.663758    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ce9689-dxhfz" event={"ID":"932dcde3-bb83-44cb-8c51-7cfa1744b0dc","Type":"ContainerStarted","Data":"4f20c496fe549836ccb06e0e786491dce27065869d6c551d365ee785fb95dba6"}
Sep 09 15:51:35.169496 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:35.169464    2553 scope.go:117] "RemoveContainer" containerID="a74daae4ba0a553e0ad5dbc450b0b9718273bf07c2593ba09afe5f713a70fd27"
Sep 09 15:51:35.179495 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:35.179468    2553 scope.go:117] "RemoveContainer" containerID="51c902e1bfe7b0c55d9ddd47992b8667e2568b6995a8e3b1156e85f943a358dd"
Sep 09 15:51:35.187639 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:35.187618    2553 scope.go:117] "RemoveContainer" containerID="b8c06d300d135a4337a276584e4d9f178733aac32ccb748a260cce70cdf791f0"
Sep 09 15:51:35.813153 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:35.813126    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ce9689-dxhfz"
Sep 09 15:51:35.996674 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:35.996575    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/932dcde3-bb83-44cb-8c51-7cfa1744b0dc-maas-api-service-ca\") pod \"932dcde3-bb83-44cb-8c51-7cfa1744b0dc\" (UID: \"932dcde3-bb83-44cb-8c51-7cfa1744b0dc\") "
Sep 09 15:51:35.997040 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:35.997007    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/932dcde3-bb83-44cb-8c51-7cfa1744b0dc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "932dcde3-bb83-44cb-8c51-7cfa1744b0dc" (UID: "932dcde3-bb83-44cb-8c51-7cfa1744b0dc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:51:36.098111 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:36.098068    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/932dcde3-bb83-44cb-8c51-7cfa1744b0dc-maas-api-service-ca\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:51:36.675949 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:36.675907    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ce9689-dxhfz" event={"ID":"932dcde3-bb83-44cb-8c51-7cfa1744b0dc","Type":"ContainerDied","Data":"4f20c496fe549836ccb06e0e786491dce27065869d6c551d365ee785fb95dba6"}
Sep 09 15:51:36.675949 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:36.675937    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ce9689-dxhfz"
Sep 09 15:51:36.676167 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:36.675967    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f20c496fe549836ccb06e0e786491dce27065869d6c551d365ee785fb95dba6"
Sep 09 15:51:48.909509 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:48.909470    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ce9689-9ff9b6fb5-ptgdt"]
Sep 09 15:51:48.910010 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:48.909756    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-ce9689-9ff9b6fb5-ptgdt" podUID="67ecd926-8585-4bd7-824d-c625cb47671c" containerName="payload-processing" containerID="cri-o://ca9f12a0ff44f8d5850eb5d9abca2e6fde116b4ede6f4332e553600af629c42d" gracePeriod=30
Sep 09 15:51:49.070060 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:49.070025    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ce9689-dxhfz"]
Sep 09 15:51:49.074975 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:49.074944    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ce9689-dxhfz"]
Sep 09 15:51:49.187055 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:49.187031    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ce9689-9ff9b6fb5-ptgdt"
Sep 09 15:51:49.231210 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:49.231170    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x55vq\" (UniqueName: \"kubernetes.io/projected/67ecd926-8585-4bd7-824d-c625cb47671c-kube-api-access-x55vq\") pod \"67ecd926-8585-4bd7-824d-c625cb47671c\" (UID: \"67ecd926-8585-4bd7-824d-c625cb47671c\") "
Sep 09 15:51:49.231398 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:49.231252    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ecd926-8585-4bd7-824d-c625cb47671c-plugins-config-volume\") pod \"67ecd926-8585-4bd7-824d-c625cb47671c\" (UID: \"67ecd926-8585-4bd7-824d-c625cb47671c\") "
Sep 09 15:51:49.231700 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:49.231671    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67ecd926-8585-4bd7-824d-c625cb47671c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "67ecd926-8585-4bd7-824d-c625cb47671c" (UID: "67ecd926-8585-4bd7-824d-c625cb47671c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:51:49.233478 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:49.233448    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67ecd926-8585-4bd7-824d-c625cb47671c-kube-api-access-x55vq" (OuterVolumeSpecName: "kube-api-access-x55vq") pod "67ecd926-8585-4bd7-824d-c625cb47671c" (UID: "67ecd926-8585-4bd7-824d-c625cb47671c"). InnerVolumeSpecName "kube-api-access-x55vq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:51:49.332594 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:49.332561    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x55vq\" (UniqueName: \"kubernetes.io/projected/67ecd926-8585-4bd7-824d-c625cb47671c-kube-api-access-x55vq\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:51:49.332594 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:49.332594    2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ecd926-8585-4bd7-824d-c625cb47671c-plugins-config-volume\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:51:49.762555 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:49.762516    2553 generic.go:356] "Generic (PLEG): container finished" podID="67ecd926-8585-4bd7-824d-c625cb47671c" containerID="ca9f12a0ff44f8d5850eb5d9abca2e6fde116b4ede6f4332e553600af629c42d" exitCode=0
Sep 09 15:51:49.762784 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:49.762627    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ce9689-9ff9b6fb5-ptgdt"
Sep 09 15:51:49.762784 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:49.762670    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ce9689-9ff9b6fb5-ptgdt" event={"ID":"67ecd926-8585-4bd7-824d-c625cb47671c","Type":"ContainerDied","Data":"ca9f12a0ff44f8d5850eb5d9abca2e6fde116b4ede6f4332e553600af629c42d"}
Sep 09 15:51:49.762784 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:49.762754    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ce9689-9ff9b6fb5-ptgdt" event={"ID":"67ecd926-8585-4bd7-824d-c625cb47671c","Type":"ContainerDied","Data":"5f0d84ae724e9fcee3c4f25d908044dfc60ad9365b3a2e9e324e77847a84bf6e"}
Sep 09 15:51:49.762978 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:49.762794    2553 scope.go:117] "RemoveContainer" containerID="ca9f12a0ff44f8d5850eb5d9abca2e6fde116b4ede6f4332e553600af629c42d"
Sep 09 15:51:49.780535 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:49.780503    2553 scope.go:117] "RemoveContainer" containerID="ca9f12a0ff44f8d5850eb5d9abca2e6fde116b4ede6f4332e553600af629c42d"
Sep 09 15:51:49.781198 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:51:49.781174    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca9f12a0ff44f8d5850eb5d9abca2e6fde116b4ede6f4332e553600af629c42d\": container with ID starting with ca9f12a0ff44f8d5850eb5d9abca2e6fde116b4ede6f4332e553600af629c42d not found: ID does not exist" containerID="ca9f12a0ff44f8d5850eb5d9abca2e6fde116b4ede6f4332e553600af629c42d"
Sep 09 15:51:49.781298 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:49.781205    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca9f12a0ff44f8d5850eb5d9abca2e6fde116b4ede6f4332e553600af629c42d"} err="failed to get container status \"ca9f12a0ff44f8d5850eb5d9abca2e6fde116b4ede6f4332e553600af629c42d\": rpc error: code = NotFound desc = could not find container \"ca9f12a0ff44f8d5850eb5d9abca2e6fde116b4ede6f4332e553600af629c42d\": container with ID starting with ca9f12a0ff44f8d5850eb5d9abca2e6fde116b4ede6f4332e553600af629c42d not found: ID does not exist"
Sep 09 15:51:49.797028 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:49.796996    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ce9689-9ff9b6fb5-ptgdt"]
Sep 09 15:51:49.800514 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:49.800485    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ce9689-9ff9b6fb5-ptgdt"]
Sep 09 15:51:50.963019 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:50.962987    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67ecd926-8585-4bd7-824d-c625cb47671c" path="/var/lib/kubelet/pods/67ecd926-8585-4bd7-824d-c625cb47671c/volumes"
Sep 09 15:51:50.963405 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:50.963340    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="932dcde3-bb83-44cb-8c51-7cfa1744b0dc" path="/var/lib/kubelet/pods/932dcde3-bb83-44cb-8c51-7cfa1744b0dc/volumes"
Sep 09 15:51:55.386689 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:55.386659    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"]
Sep 09 15:51:55.387100 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:55.386921    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv" podUID="79946caa-e46d-4320-a5ec-80ad8a2c7bd9" containerName="istio-proxy" containerID="cri-o://40beab8a1b12c716de7200bdf99b5a7a6f220878b0b6817d4a14fcfc4b472555" gracePeriod=30
Sep 09 15:51:55.730477 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:55.730446    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ce3053-pjjbk"]
Sep 09 15:51:55.730852 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:55.730839    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="932dcde3-bb83-44cb-8c51-7cfa1744b0dc" containerName="revoke-keys"
Sep 09 15:51:55.730907 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:55.730854    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="932dcde3-bb83-44cb-8c51-7cfa1744b0dc" containerName="revoke-keys"
Sep 09 15:51:55.730907 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:55.730877    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67ecd926-8585-4bd7-824d-c625cb47671c" containerName="payload-processing"
Sep 09 15:51:55.730907 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:55.730883    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="67ecd926-8585-4bd7-824d-c625cb47671c" containerName="payload-processing"
Sep 09 15:51:55.730999 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:55.730942    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="67ecd926-8585-4bd7-824d-c625cb47671c" containerName="payload-processing"
Sep 09 15:51:55.730999 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:55.730952    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="932dcde3-bb83-44cb-8c51-7cfa1744b0dc" containerName="revoke-keys"
Sep 09 15:51:55.735551 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:55.735532    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ce3053-pjjbk"
Sep 09 15:51:55.738235 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:55.738205    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 15:51:55.738371 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:55.738238    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2zlsf\""
Sep 09 15:51:55.741541 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:55.741512    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ce3053-pjjbk"]
Sep 09 15:51:55.795496 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:55.795462    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9df8b10-f450-4fd9-8588-32c7c608ae71-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-ce3053-pjjbk\" (UID: \"d9df8b10-f450-4fd9-8588-32c7c608ae71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ce3053-pjjbk"
Sep 09 15:51:55.897093 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:55.897059    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9df8b10-f450-4fd9-8588-32c7c608ae71-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-ce3053-pjjbk\" (UID: \"d9df8b10-f450-4fd9-8588-32c7c608ae71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ce3053-pjjbk"
Sep 09 15:51:55.897671 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:55.897652    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9df8b10-f450-4fd9-8588-32c7c608ae71-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-ce3053-pjjbk\" (UID: \"d9df8b10-f450-4fd9-8588-32c7c608ae71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ce3053-pjjbk"
Sep 09 15:51:56.048925 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:56.048842    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ce3053-pjjbk"
Sep 09 15:51:56.326047 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:56.325967    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ce3053-pjjbk"]
Sep 09 15:51:56.329116 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:51:56.329085    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9df8b10_f450_4fd9_8588_32c7c608ae71.slice/crio-e3fec850450a020386d08c8c85afe58036e3c597dbb6028e53126c2dc3de9398 WatchSource:0}: Error finding container e3fec850450a020386d08c8c85afe58036e3c597dbb6028e53126c2dc3de9398: Status 404 returned error can't find the container with id e3fec850450a020386d08c8c85afe58036e3c597dbb6028e53126c2dc3de9398
Sep 09 15:51:56.807230 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:56.807191    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ce3053-pjjbk" event={"ID":"d9df8b10-f450-4fd9-8588-32c7c608ae71","Type":"ContainerStarted","Data":"b3db36d0d490828061ef133b2ff954b8a091b6acddf64fe70f4bca3742d076e7"}
Sep 09 15:51:56.807620 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:56.807254    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ce3053-pjjbk" event={"ID":"d9df8b10-f450-4fd9-8588-32c7c608ae71","Type":"ContainerStarted","Data":"e3fec850450a020386d08c8c85afe58036e3c597dbb6028e53126c2dc3de9398"}
Sep 09 15:51:56.822583 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:56.822529    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ce3053-pjjbk" podStartSLOduration=1.8225129 podStartE2EDuration="1.8225129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:51:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:51:56.821715528 +0000 UTC m=+1882.420610173" watchObservedRunningTime="2026-09-09 15:51:56.8225129 +0000 UTC m=+1882.421407545"
Sep 09 15:51:57.814610 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:57.814572    2553 generic.go:356] "Generic (PLEG): container finished" podID="d9df8b10-f450-4fd9-8588-32c7c608ae71" containerID="b3db36d0d490828061ef133b2ff954b8a091b6acddf64fe70f4bca3742d076e7" exitCode=0
Sep 09 15:51:57.815085 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:57.814656    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ce3053-pjjbk" event={"ID":"d9df8b10-f450-4fd9-8588-32c7c608ae71","Type":"ContainerDied","Data":"b3db36d0d490828061ef133b2ff954b8a091b6acddf64fe70f4bca3742d076e7"}
Sep 09 15:51:58.963399 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:58.963367    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ce3053-pjjbk"
Sep 09 15:51:59.031029 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:59.030995    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9df8b10-f450-4fd9-8588-32c7c608ae71-maas-api-service-ca\") pod \"d9df8b10-f450-4fd9-8588-32c7c608ae71\" (UID: \"d9df8b10-f450-4fd9-8588-32c7c608ae71\") "
Sep 09 15:51:59.031368 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:59.031342    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9df8b10-f450-4fd9-8588-32c7c608ae71-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d9df8b10-f450-4fd9-8588-32c7c608ae71" (UID: "d9df8b10-f450-4fd9-8588-32c7c608ae71"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:51:59.132532 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:59.132445    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9df8b10-f450-4fd9-8588-32c7c608ae71-maas-api-service-ca\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:51:59.826668 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:59.826640    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ce3053-pjjbk"
Sep 09 15:51:59.826668 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:59.826652    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ce3053-pjjbk" event={"ID":"d9df8b10-f450-4fd9-8588-32c7c608ae71","Type":"ContainerDied","Data":"e3fec850450a020386d08c8c85afe58036e3c597dbb6028e53126c2dc3de9398"}
Sep 09 15:51:59.826915 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:51:59.826702    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3fec850450a020386d08c8c85afe58036e3c597dbb6028e53126c2dc3de9398"
Sep 09 15:52:00.646595 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.646569    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"
Sep 09 15:52:00.754392 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.754361    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-istio-podinfo\") pod \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\" (UID: \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\") "
Sep 09 15:52:00.754605 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.754414    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-credential-socket\") pod \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\" (UID: \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\") "
Sep 09 15:52:00.754605 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.754460    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-istio-token\") pod \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\" (UID: \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\") "
Sep 09 15:52:00.754605 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.754495    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-workload-socket\") pod \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\" (UID: \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\") "
Sep 09 15:52:00.754605 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.754520    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-istio-data\") pod \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\" (UID: \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\") "
Sep 09 15:52:00.754605 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.754543    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-istiod-ca-cert\") pod \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\" (UID: \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\") "
Sep 09 15:52:00.754605 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.754590    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n77tn\" (UniqueName: \"kubernetes.io/projected/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-kube-api-access-n77tn\") pod \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\" (UID: \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\") "
Sep 09 15:52:00.755188 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.754617    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-workload-certs\") pod \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\" (UID: \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\") "
Sep 09 15:52:00.755188 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.754684    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-istio-envoy\") pod \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\" (UID: \"79946caa-e46d-4320-a5ec-80ad8a2c7bd9\") "
Sep 09 15:52:00.755188 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.754796    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "79946caa-e46d-4320-a5ec-80ad8a2c7bd9" (UID: "79946caa-e46d-4320-a5ec-80ad8a2c7bd9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:52:00.755188 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.754930    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "79946caa-e46d-4320-a5ec-80ad8a2c7bd9" (UID: "79946caa-e46d-4320-a5ec-80ad8a2c7bd9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:52:00.755188 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.754954    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-istio-data" (OuterVolumeSpecName: "istio-data") pod "79946caa-e46d-4320-a5ec-80ad8a2c7bd9" (UID: "79946caa-e46d-4320-a5ec-80ad8a2c7bd9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:52:00.755471 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.755190    2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-credential-socket\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:52:00.755471 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.755212    2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-workload-socket\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:52:00.755471 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.755227    2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-istio-data\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:52:00.755471 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.755192    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "79946caa-e46d-4320-a5ec-80ad8a2c7bd9" (UID: "79946caa-e46d-4320-a5ec-80ad8a2c7bd9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:52:00.755471 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.755411    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "79946caa-e46d-4320-a5ec-80ad8a2c7bd9" (UID: "79946caa-e46d-4320-a5ec-80ad8a2c7bd9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:52:00.756817 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.756785    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-istio-token" (OuterVolumeSpecName: "istio-token") pod "79946caa-e46d-4320-a5ec-80ad8a2c7bd9" (UID: "79946caa-e46d-4320-a5ec-80ad8a2c7bd9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:52:00.756936 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.756885    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-kube-api-access-n77tn" (OuterVolumeSpecName: "kube-api-access-n77tn") pod "79946caa-e46d-4320-a5ec-80ad8a2c7bd9" (UID: "79946caa-e46d-4320-a5ec-80ad8a2c7bd9"). InnerVolumeSpecName "kube-api-access-n77tn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:52:00.756936 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.756891    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "79946caa-e46d-4320-a5ec-80ad8a2c7bd9" (UID: "79946caa-e46d-4320-a5ec-80ad8a2c7bd9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 15:52:00.757042 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.757024    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "79946caa-e46d-4320-a5ec-80ad8a2c7bd9" (UID: "79946caa-e46d-4320-a5ec-80ad8a2c7bd9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:52:00.835023 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.834989    2553 generic.go:356] "Generic (PLEG): container finished" podID="79946caa-e46d-4320-a5ec-80ad8a2c7bd9" containerID="40beab8a1b12c716de7200bdf99b5a7a6f220878b0b6817d4a14fcfc4b472555" exitCode=0
Sep 09 15:52:00.835214 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.835081    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"
Sep 09 15:52:00.835214 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.835080    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv" event={"ID":"79946caa-e46d-4320-a5ec-80ad8a2c7bd9","Type":"ContainerDied","Data":"40beab8a1b12c716de7200bdf99b5a7a6f220878b0b6817d4a14fcfc4b472555"}
Sep 09 15:52:00.835214 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.835148    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv" event={"ID":"79946caa-e46d-4320-a5ec-80ad8a2c7bd9","Type":"ContainerDied","Data":"8007e64d9a45b69be930ba9e33912a27ff57bc059200a8dd3dd5d7b05d3f3f28"}
Sep 09 15:52:00.835214 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.835181    2553 scope.go:117] "RemoveContainer" containerID="40beab8a1b12c716de7200bdf99b5a7a6f220878b0b6817d4a14fcfc4b472555"
Sep 09 15:52:00.846855 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.846835    2553 scope.go:117] "RemoveContainer" containerID="a65ce739ef66a78f818233f78d1198d2fc6301ef17918dd6eea9eb9869820593"
Sep 09 15:52:00.855479 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.855460    2553 scope.go:117] "RemoveContainer" containerID="40beab8a1b12c716de7200bdf99b5a7a6f220878b0b6817d4a14fcfc4b472555"
Sep 09 15:52:00.855817 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:52:00.855797    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40beab8a1b12c716de7200bdf99b5a7a6f220878b0b6817d4a14fcfc4b472555\": container with ID starting with 40beab8a1b12c716de7200bdf99b5a7a6f220878b0b6817d4a14fcfc4b472555 not found: ID does not exist" containerID="40beab8a1b12c716de7200bdf99b5a7a6f220878b0b6817d4a14fcfc4b472555"
Sep 09 15:52:00.855907 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.855824    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40beab8a1b12c716de7200bdf99b5a7a6f220878b0b6817d4a14fcfc4b472555"} err="failed to get container status \"40beab8a1b12c716de7200bdf99b5a7a6f220878b0b6817d4a14fcfc4b472555\": rpc error: code = NotFound desc = could not find container \"40beab8a1b12c716de7200bdf99b5a7a6f220878b0b6817d4a14fcfc4b472555\": container with ID starting with 40beab8a1b12c716de7200bdf99b5a7a6f220878b0b6817d4a14fcfc4b472555 not found: ID does not exist"
Sep 09 15:52:00.855907 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.855842    2553 scope.go:117] "RemoveContainer" containerID="a65ce739ef66a78f818233f78d1198d2fc6301ef17918dd6eea9eb9869820593"
Sep 09 15:52:00.856028 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.855939    2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-istio-envoy\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:52:00.856028 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.855961    2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-istio-podinfo\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:52:00.856028 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.855983    2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-istio-token\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:52:00.856028 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.855997    2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-istiod-ca-cert\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:52:00.856028 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.856008    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n77tn\" (UniqueName: \"kubernetes.io/projected/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-kube-api-access-n77tn\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:52:00.856028 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.856024    2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79946caa-e46d-4320-a5ec-80ad8a2c7bd9-workload-certs\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:52:00.856407 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:52:00.856176    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a65ce739ef66a78f818233f78d1198d2fc6301ef17918dd6eea9eb9869820593\": container with ID starting with a65ce739ef66a78f818233f78d1198d2fc6301ef17918dd6eea9eb9869820593 not found: ID does not exist" containerID="a65ce739ef66a78f818233f78d1198d2fc6301ef17918dd6eea9eb9869820593"
Sep 09 15:52:00.856407 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.856207    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a65ce739ef66a78f818233f78d1198d2fc6301ef17918dd6eea9eb9869820593"} err="failed to get container status \"a65ce739ef66a78f818233f78d1198d2fc6301ef17918dd6eea9eb9869820593\": rpc error: code = NotFound desc = could not find container \"a65ce739ef66a78f818233f78d1198d2fc6301ef17918dd6eea9eb9869820593\": container with ID starting with a65ce739ef66a78f818233f78d1198d2fc6301ef17918dd6eea9eb9869820593 not found: ID does not exist"
Sep 09 15:52:00.859272 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.859251    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"]
Sep 09 15:52:00.864912 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.864892    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-ce9689-openshift-default-7946dc57c4-xk5rv"]
Sep 09 15:52:00.963066 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:00.963028    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79946caa-e46d-4320-a5ec-80ad8a2c7bd9" path="/var/lib/kubelet/pods/79946caa-e46d-4320-a5ec-80ad8a2c7bd9/volumes"
Sep 09 15:52:06.243338 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:06.242552    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce3053-7d64fc88d6-rmk9b"]
Sep 09 15:52:06.243338 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:06.242814    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce3053-7d64fc88d6-rmk9b" podUID="8f9f978e-3d58-4841-99f4-d1f26ff35f3a" containerName="payload-pre-processing" containerID="cri-o://d5641858d020b5a0fb204c4b5630b2ec76aeaa23edea23a8c1a53162b237cce6" gracePeriod=30
Sep 09 15:52:06.387095 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:06.387050    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ce3053-pjjbk"]
Sep 09 15:52:06.391047 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:06.391017    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ce3053-pjjbk"]
Sep 09 15:52:06.508371 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:06.508301    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce3053-7d64fc88d6-rmk9b"
Sep 09 15:52:06.616839 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:06.616793    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f9f978e-3d58-4841-99f4-d1f26ff35f3a-plugins-config-volume\") pod \"8f9f978e-3d58-4841-99f4-d1f26ff35f3a\" (UID: \"8f9f978e-3d58-4841-99f4-d1f26ff35f3a\") "
Sep 09 15:52:06.617053 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:06.616948    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qz2qg\" (UniqueName: \"kubernetes.io/projected/8f9f978e-3d58-4841-99f4-d1f26ff35f3a-kube-api-access-qz2qg\") pod \"8f9f978e-3d58-4841-99f4-d1f26ff35f3a\" (UID: \"8f9f978e-3d58-4841-99f4-d1f26ff35f3a\") "
Sep 09 15:52:06.617269 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:06.617241    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f9f978e-3d58-4841-99f4-d1f26ff35f3a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8f9f978e-3d58-4841-99f4-d1f26ff35f3a" (UID: "8f9f978e-3d58-4841-99f4-d1f26ff35f3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:52:06.619014 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:06.618984    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f9f978e-3d58-4841-99f4-d1f26ff35f3a-kube-api-access-qz2qg" (OuterVolumeSpecName: "kube-api-access-qz2qg") pod "8f9f978e-3d58-4841-99f4-d1f26ff35f3a" (UID: "8f9f978e-3d58-4841-99f4-d1f26ff35f3a"). InnerVolumeSpecName "kube-api-access-qz2qg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:52:06.717704 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:06.717661    2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f9f978e-3d58-4841-99f4-d1f26ff35f3a-plugins-config-volume\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:52:06.717704 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:06.717697    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz2qg\" (UniqueName: \"kubernetes.io/projected/8f9f978e-3d58-4841-99f4-d1f26ff35f3a-kube-api-access-qz2qg\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:52:06.872635 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:06.872538    2553 generic.go:356] "Generic (PLEG): container finished" podID="8f9f978e-3d58-4841-99f4-d1f26ff35f3a" containerID="d5641858d020b5a0fb204c4b5630b2ec76aeaa23edea23a8c1a53162b237cce6" exitCode=0
Sep 09 15:52:06.872635 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:06.872580    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce3053-7d64fc88d6-rmk9b"
Sep 09 15:52:06.872893 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:06.872624    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce3053-7d64fc88d6-rmk9b" event={"ID":"8f9f978e-3d58-4841-99f4-d1f26ff35f3a","Type":"ContainerDied","Data":"d5641858d020b5a0fb204c4b5630b2ec76aeaa23edea23a8c1a53162b237cce6"}
Sep 09 15:52:06.872893 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:06.872683    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce3053-7d64fc88d6-rmk9b" event={"ID":"8f9f978e-3d58-4841-99f4-d1f26ff35f3a","Type":"ContainerDied","Data":"5f4a0e09ab8f2e64644508d73adf5f2924a3d3ec2494d7cf1c2a45881adc0fab"}
Sep 09 15:52:06.872893 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:06.872713    2553 scope.go:117] "RemoveContainer" containerID="d5641858d020b5a0fb204c4b5630b2ec76aeaa23edea23a8c1a53162b237cce6"
Sep 09 15:52:06.882254 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:06.882237    2553 scope.go:117] "RemoveContainer" containerID="d5641858d020b5a0fb204c4b5630b2ec76aeaa23edea23a8c1a53162b237cce6"
Sep 09 15:52:06.882530 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:52:06.882512    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5641858d020b5a0fb204c4b5630b2ec76aeaa23edea23a8c1a53162b237cce6\": container with ID starting with d5641858d020b5a0fb204c4b5630b2ec76aeaa23edea23a8c1a53162b237cce6 not found: ID does not exist" containerID="d5641858d020b5a0fb204c4b5630b2ec76aeaa23edea23a8c1a53162b237cce6"
Sep 09 15:52:06.882587 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:06.882540    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5641858d020b5a0fb204c4b5630b2ec76aeaa23edea23a8c1a53162b237cce6"} err="failed to get container status \"d5641858d020b5a0fb204c4b5630b2ec76aeaa23edea23a8c1a53162b237cce6\": rpc error: code = NotFound desc = could not find container \"d5641858d020b5a0fb204c4b5630b2ec76aeaa23edea23a8c1a53162b237cce6\": container with ID starting with d5641858d020b5a0fb204c4b5630b2ec76aeaa23edea23a8c1a53162b237cce6 not found: ID does not exist"
Sep 09 15:52:06.895717 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:06.895688    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce3053-7d64fc88d6-rmk9b"]
Sep 09 15:52:06.900561 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:06.900529    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce3053-7d64fc88d6-rmk9b"]
Sep 09 15:52:06.963417 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:06.963379    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f9f978e-3d58-4841-99f4-d1f26ff35f3a" path="/var/lib/kubelet/pods/8f9f978e-3d58-4841-99f4-d1f26ff35f3a/volumes"
Sep 09 15:52:06.963761 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:06.963747    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9df8b10-f450-4fd9-8588-32c7c608ae71" path="/var/lib/kubelet/pods/d9df8b10-f450-4fd9-8588-32c7c608ae71/volumes"
Sep 09 15:52:13.072494 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:13.072455    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-081ed3-kdv5h"]
Sep 09 15:52:13.072897 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:13.072847    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79946caa-e46d-4320-a5ec-80ad8a2c7bd9" containerName="istio-proxy"
Sep 09 15:52:13.072897 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:13.072861    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="79946caa-e46d-4320-a5ec-80ad8a2c7bd9" containerName="istio-proxy"
Sep 09 15:52:13.072897 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:13.072873    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79946caa-e46d-4320-a5ec-80ad8a2c7bd9" containerName="istio-proxy"
Sep 09 15:52:13.072897 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:13.072879    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="79946caa-e46d-4320-a5ec-80ad8a2c7bd9" containerName="istio-proxy"
Sep 09 15:52:13.072897 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:13.072895    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9df8b10-f450-4fd9-8588-32c7c608ae71" containerName="revoke-keys"
Sep 09 15:52:13.072897 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:13.072901    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9df8b10-f450-4fd9-8588-32c7c608ae71" containerName="revoke-keys"
Sep 09 15:52:13.073094 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:13.072911    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f9f978e-3d58-4841-99f4-d1f26ff35f3a" containerName="payload-pre-processing"
Sep 09 15:52:13.073094 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:13.072917    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f9f978e-3d58-4841-99f4-d1f26ff35f3a" containerName="payload-pre-processing"
Sep 09 15:52:13.073094 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:13.072975    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="79946caa-e46d-4320-a5ec-80ad8a2c7bd9" containerName="istio-proxy"
Sep 09 15:52:13.073094 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:13.072983    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9df8b10-f450-4fd9-8588-32c7c608ae71" containerName="revoke-keys"
Sep 09 15:52:13.073094 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:13.072990    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f9f978e-3d58-4841-99f4-d1f26ff35f3a" containerName="payload-pre-processing"
Sep 09 15:52:13.076018 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:13.076001    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-081ed3-kdv5h"
Sep 09 15:52:13.078476 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:13.078452    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2zlsf\""
Sep 09 15:52:13.078726 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:13.078706    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 15:52:13.083205 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:13.083176    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-081ed3-kdv5h"]
Sep 09 15:52:13.185349 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:13.185317    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a4fbcf2-bbb3-471c-b11f-1673dd438efa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-081ed3-kdv5h\" (UID: \"5a4fbcf2-bbb3-471c-b11f-1673dd438efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-081ed3-kdv5h"
Sep 09 15:52:13.286169 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:13.286134    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a4fbcf2-bbb3-471c-b11f-1673dd438efa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-081ed3-kdv5h\" (UID: \"5a4fbcf2-bbb3-471c-b11f-1673dd438efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-081ed3-kdv5h"
Sep 09 15:52:13.286847 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:13.286821    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a4fbcf2-bbb3-471c-b11f-1673dd438efa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-081ed3-kdv5h\" (UID: \"5a4fbcf2-bbb3-471c-b11f-1673dd438efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-081ed3-kdv5h"
Sep 09 15:52:13.387637 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:13.387550    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-081ed3-kdv5h"
Sep 09 15:52:13.659276 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:13.659251    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-081ed3-kdv5h"]
Sep 09 15:52:13.661295 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:52:13.661261    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a4fbcf2_bbb3_471c_b11f_1673dd438efa.slice/crio-525a137839a759882113f5df7338fc3d787f52dd12bd15e16f4b668ad9d9e61c WatchSource:0}: Error finding container 525a137839a759882113f5df7338fc3d787f52dd12bd15e16f4b668ad9d9e61c: Status 404 returned error can't find the container with id 525a137839a759882113f5df7338fc3d787f52dd12bd15e16f4b668ad9d9e61c
Sep 09 15:52:13.914634 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:13.914540    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-081ed3-kdv5h" event={"ID":"5a4fbcf2-bbb3-471c-b11f-1673dd438efa","Type":"ContainerStarted","Data":"525a137839a759882113f5df7338fc3d787f52dd12bd15e16f4b668ad9d9e61c"}
Sep 09 15:52:14.921427 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:14.921396    2553 generic.go:356] "Generic (PLEG): container finished" podID="5a4fbcf2-bbb3-471c-b11f-1673dd438efa" containerID="9393a38f28d43b4cb22dd216f0662e2431bbb7dd953ab902ad7679e3d271566d" exitCode=0
Sep 09 15:52:14.921832 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:14.921475    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-081ed3-kdv5h" event={"ID":"5a4fbcf2-bbb3-471c-b11f-1673dd438efa","Type":"ContainerDied","Data":"9393a38f28d43b4cb22dd216f0662e2431bbb7dd953ab902ad7679e3d271566d"}
Sep 09 15:52:16.063634 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:16.063608    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-081ed3-kdv5h"
Sep 09 15:52:16.113699 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:16.113661    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a4fbcf2-bbb3-471c-b11f-1673dd438efa-maas-api-service-ca\") pod \"5a4fbcf2-bbb3-471c-b11f-1673dd438efa\" (UID: \"5a4fbcf2-bbb3-471c-b11f-1673dd438efa\") "
Sep 09 15:52:16.114062 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:16.114037    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a4fbcf2-bbb3-471c-b11f-1673dd438efa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5a4fbcf2-bbb3-471c-b11f-1673dd438efa" (UID: "5a4fbcf2-bbb3-471c-b11f-1673dd438efa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:52:16.215665 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:16.215563    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a4fbcf2-bbb3-471c-b11f-1673dd438efa-maas-api-service-ca\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:52:16.933531 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:16.933500    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-081ed3-kdv5h"
Sep 09 15:52:16.933728 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:16.933493    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-081ed3-kdv5h" event={"ID":"5a4fbcf2-bbb3-471c-b11f-1673dd438efa","Type":"ContainerDied","Data":"525a137839a759882113f5df7338fc3d787f52dd12bd15e16f4b668ad9d9e61c"}
Sep 09 15:52:16.933728 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:16.933642    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="525a137839a759882113f5df7338fc3d787f52dd12bd15e16f4b668ad9d9e61c"
Sep 09 15:52:23.512567 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:23.512500    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-081ed3-29816145-qk6nx"]
Sep 09 15:52:23.515214 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:23.515191    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-081ed3-29816145-qk6nx"]
Sep 09 15:52:23.544027 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:23.543990    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-081ed3-6b74b9d8f-f4zz9"]
Sep 09 15:52:23.544319 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:23.544254    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-081ed3-6b74b9d8f-f4zz9" podUID="e1eca92b-d2aa-4f69-85df-4a67708a08a9" containerName="payload-processing" containerID="cri-o://81aac16796328cbbe91d7b375db3465bb21907b2f207a7b67cad85f1688cac3b" gracePeriod=30
Sep 09 15:52:23.685892 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:23.685861    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-081ed3-kdv5h"]
Sep 09 15:52:23.690059 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:23.690031    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-081ed3-kdv5h"]
Sep 09 15:52:23.811245 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:23.811215    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-081ed3-6b74b9d8f-f4zz9"
Sep 09 15:52:23.891573 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:23.891540    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1eca92b-d2aa-4f69-85df-4a67708a08a9-plugins-config-volume\") pod \"e1eca92b-d2aa-4f69-85df-4a67708a08a9\" (UID: \"e1eca92b-d2aa-4f69-85df-4a67708a08a9\") "
Sep 09 15:52:23.891787 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:23.891592    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7n6g\" (UniqueName: \"kubernetes.io/projected/e1eca92b-d2aa-4f69-85df-4a67708a08a9-kube-api-access-l7n6g\") pod \"e1eca92b-d2aa-4f69-85df-4a67708a08a9\" (UID: \"e1eca92b-d2aa-4f69-85df-4a67708a08a9\") "
Sep 09 15:52:23.891948 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:23.891925    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1eca92b-d2aa-4f69-85df-4a67708a08a9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e1eca92b-d2aa-4f69-85df-4a67708a08a9" (UID: "e1eca92b-d2aa-4f69-85df-4a67708a08a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:52:23.893632 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:23.893608    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1eca92b-d2aa-4f69-85df-4a67708a08a9-kube-api-access-l7n6g" (OuterVolumeSpecName: "kube-api-access-l7n6g") pod "e1eca92b-d2aa-4f69-85df-4a67708a08a9" (UID: "e1eca92b-d2aa-4f69-85df-4a67708a08a9"). InnerVolumeSpecName "kube-api-access-l7n6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:52:23.975716 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:23.975677    2553 generic.go:356] "Generic (PLEG): container finished" podID="e1eca92b-d2aa-4f69-85df-4a67708a08a9" containerID="81aac16796328cbbe91d7b375db3465bb21907b2f207a7b67cad85f1688cac3b" exitCode=0
Sep 09 15:52:23.975716 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:23.975727    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-081ed3-6b74b9d8f-f4zz9"
Sep 09 15:52:23.975954 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:23.975753    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-081ed3-6b74b9d8f-f4zz9" event={"ID":"e1eca92b-d2aa-4f69-85df-4a67708a08a9","Type":"ContainerDied","Data":"81aac16796328cbbe91d7b375db3465bb21907b2f207a7b67cad85f1688cac3b"}
Sep 09 15:52:23.975954 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:23.975815    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-081ed3-6b74b9d8f-f4zz9" event={"ID":"e1eca92b-d2aa-4f69-85df-4a67708a08a9","Type":"ContainerDied","Data":"fe6f3f3224620c819b9090c7169fe4974a4fd40389fde62cfd2cee0a2bf948f4"}
Sep 09 15:52:23.975954 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:23.975840    2553 scope.go:117] "RemoveContainer" containerID="81aac16796328cbbe91d7b375db3465bb21907b2f207a7b67cad85f1688cac3b"
Sep 09 15:52:23.986001 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:23.985982    2553 scope.go:117] "RemoveContainer" containerID="81aac16796328cbbe91d7b375db3465bb21907b2f207a7b67cad85f1688cac3b"
Sep 09 15:52:23.986291 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:52:23.986271    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81aac16796328cbbe91d7b375db3465bb21907b2f207a7b67cad85f1688cac3b\": container with ID starting with 81aac16796328cbbe91d7b375db3465bb21907b2f207a7b67cad85f1688cac3b not found: ID does not exist" containerID="81aac16796328cbbe91d7b375db3465bb21907b2f207a7b67cad85f1688cac3b"
Sep 09 15:52:23.986330 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:23.986301    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81aac16796328cbbe91d7b375db3465bb21907b2f207a7b67cad85f1688cac3b"} err="failed to get container status \"81aac16796328cbbe91d7b375db3465bb21907b2f207a7b67cad85f1688cac3b\": rpc error: code = NotFound desc = could not find container \"81aac16796328cbbe91d7b375db3465bb21907b2f207a7b67cad85f1688cac3b\": container with ID starting with 81aac16796328cbbe91d7b375db3465bb21907b2f207a7b67cad85f1688cac3b not found: ID does not exist"
Sep 09 15:52:23.993018 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:23.992994    2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1eca92b-d2aa-4f69-85df-4a67708a08a9-plugins-config-volume\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:52:23.993018 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:23.993019    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7n6g\" (UniqueName: \"kubernetes.io/projected/e1eca92b-d2aa-4f69-85df-4a67708a08a9-kube-api-access-l7n6g\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:52:23.999511 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:23.999489    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-081ed3-6b74b9d8f-f4zz9"]
Sep 09 15:52:24.003442 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:24.003419    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-081ed3-6b74b9d8f-f4zz9"]
Sep 09 15:52:24.962688 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:24.962654    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38de0606-7eb9-4429-8c05-2d9a8fa23465" path="/var/lib/kubelet/pods/38de0606-7eb9-4429-8c05-2d9a8fa23465/volumes"
Sep 09 15:52:24.963079 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:24.962977    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a4fbcf2-bbb3-471c-b11f-1673dd438efa" path="/var/lib/kubelet/pods/5a4fbcf2-bbb3-471c-b11f-1673dd438efa/volumes"
Sep 09 15:52:24.963323 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:24.963309    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1eca92b-d2aa-4f69-85df-4a67708a08a9" path="/var/lib/kubelet/pods/e1eca92b-d2aa-4f69-85df-4a67708a08a9/volumes"
Sep 09 15:52:30.625898 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:30.625865    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"]
Sep 09 15:52:30.626419 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:30.626132    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx" podUID="cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b" containerName="istio-proxy" containerID="cri-o://18fb07461c9bc8d8ab91b77ae5af9db257101c11aeea06b47466d0e6a6292da8" gracePeriod=30
Sep 09 15:52:32.830718 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:32.830677    2553 patch_prober.go:28] interesting pod/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 15:52:32.831171 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:32.830765    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx" podUID="cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 15:52:35.219582 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:35.219553    2553 scope.go:117] "RemoveContainer" containerID="c170b7a062ef4cf28e736f0a75135719ecaada8258ed67c315be586aafa7866d"
Sep 09 15:52:35.228092 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:35.228066    2553 scope.go:117] "RemoveContainer" containerID="fc0e1876aa5f5b4eb07c2a97af807623935f322d989e0458334833e1e24a7e42"
Sep 09 15:52:35.878329 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:35.878300    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"
Sep 09 15:52:36.012147 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.012108    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-istio-token\") pod \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\" (UID: \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\") "
Sep 09 15:52:36.012338 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.012190    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-workload-certs\") pod \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\" (UID: \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\") "
Sep 09 15:52:36.012338 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.012217    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-istio-envoy\") pod \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\" (UID: \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\") "
Sep 09 15:52:36.012338 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.012249    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twqlx\" (UniqueName: \"kubernetes.io/projected/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-kube-api-access-twqlx\") pod \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\" (UID: \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\") "
Sep 09 15:52:36.012338 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.012302    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-istiod-ca-cert\") pod \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\" (UID: \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\") "
Sep 09 15:52:36.012338 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.012328    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-istio-data\") pod \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\" (UID: \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\") "
Sep 09 15:52:36.012602 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.012357    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-workload-socket\") pod \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\" (UID: \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\") "
Sep 09 15:52:36.012602 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.012384    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-istio-podinfo\") pod \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\" (UID: \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\") "
Sep 09 15:52:36.012602 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.012416    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-credential-socket\") pod \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\" (UID: \"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b\") "
Sep 09 15:52:36.012602 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.012534    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b" (UID: "cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:52:36.013030 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.012687    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-istio-data" (OuterVolumeSpecName: "istio-data") pod "cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b" (UID: "cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:52:36.013148 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.012808    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b" (UID: "cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:52:36.013148 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.012879    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b" (UID: "cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:52:36.013148 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.012780    2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-workload-certs\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:52:36.013148 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.012990    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b" (UID: "cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:52:36.014577 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.014543    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-kube-api-access-twqlx" (OuterVolumeSpecName: "kube-api-access-twqlx") pod "cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b" (UID: "cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b"). InnerVolumeSpecName "kube-api-access-twqlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:52:36.014688 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.014621    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b" (UID: "cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 15:52:36.014849 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.014831    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b" (UID: "cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:52:36.014909 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.014845    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-istio-token" (OuterVolumeSpecName: "istio-token") pod "cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b" (UID: "cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:52:36.048353 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.048316    2553 generic.go:356] "Generic (PLEG): container finished" podID="cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b" containerID="18fb07461c9bc8d8ab91b77ae5af9db257101c11aeea06b47466d0e6a6292da8" exitCode=0
Sep 09 15:52:36.048526 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.048394    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"
Sep 09 15:52:36.048526 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.048400    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx" event={"ID":"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b","Type":"ContainerDied","Data":"18fb07461c9bc8d8ab91b77ae5af9db257101c11aeea06b47466d0e6a6292da8"}
Sep 09 15:52:36.048526 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.048456    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx" event={"ID":"cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b","Type":"ContainerDied","Data":"70be6a131e0e763d9467f6b392110e01be96a8c382967f8f1368dd91eb747f35"}
Sep 09 15:52:36.048526 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.048485    2553 scope.go:117] "RemoveContainer" containerID="18fb07461c9bc8d8ab91b77ae5af9db257101c11aeea06b47466d0e6a6292da8"
Sep 09 15:52:36.057989 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.057970    2553 scope.go:117] "RemoveContainer" containerID="e77362da4234b47af04d29e17c383cc0b6f299475e9508a68a9ddbd65f961242"
Sep 09 15:52:36.066328 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.066310    2553 scope.go:117] "RemoveContainer" containerID="18fb07461c9bc8d8ab91b77ae5af9db257101c11aeea06b47466d0e6a6292da8"
Sep 09 15:52:36.066622 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:52:36.066598    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18fb07461c9bc8d8ab91b77ae5af9db257101c11aeea06b47466d0e6a6292da8\": container with ID starting with 18fb07461c9bc8d8ab91b77ae5af9db257101c11aeea06b47466d0e6a6292da8 not found: ID does not exist" containerID="18fb07461c9bc8d8ab91b77ae5af9db257101c11aeea06b47466d0e6a6292da8"
Sep 09 15:52:36.066724 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.066628    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18fb07461c9bc8d8ab91b77ae5af9db257101c11aeea06b47466d0e6a6292da8"} err="failed to get container status \"18fb07461c9bc8d8ab91b77ae5af9db257101c11aeea06b47466d0e6a6292da8\": rpc error: code = NotFound desc = could not find container \"18fb07461c9bc8d8ab91b77ae5af9db257101c11aeea06b47466d0e6a6292da8\": container with ID starting with 18fb07461c9bc8d8ab91b77ae5af9db257101c11aeea06b47466d0e6a6292da8 not found: ID does not exist"
Sep 09 15:52:36.066724 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.066646    2553 scope.go:117] "RemoveContainer" containerID="e77362da4234b47af04d29e17c383cc0b6f299475e9508a68a9ddbd65f961242"
Sep 09 15:52:36.066946 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:52:36.066927    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e77362da4234b47af04d29e17c383cc0b6f299475e9508a68a9ddbd65f961242\": container with ID starting with e77362da4234b47af04d29e17c383cc0b6f299475e9508a68a9ddbd65f961242 not found: ID does not exist" containerID="e77362da4234b47af04d29e17c383cc0b6f299475e9508a68a9ddbd65f961242"
Sep 09 15:52:36.067004 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.066955    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e77362da4234b47af04d29e17c383cc0b6f299475e9508a68a9ddbd65f961242"} err="failed to get container status \"e77362da4234b47af04d29e17c383cc0b6f299475e9508a68a9ddbd65f961242\": rpc error: code = NotFound desc = could not find container \"e77362da4234b47af04d29e17c383cc0b6f299475e9508a68a9ddbd65f961242\": container with ID starting with e77362da4234b47af04d29e17c383cc0b6f299475e9508a68a9ddbd65f961242 not found: ID does not exist"
Sep 09 15:52:36.071381 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.071359    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"]
Sep 09 15:52:36.075103 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.075081    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-081ed3-openshift-default-7447c8bc94-c5hjx"]
Sep 09 15:52:36.114385 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.114350    2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-istiod-ca-cert\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:52:36.114385 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.114381    2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-istio-data\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:52:36.114385 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.114394    2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-workload-socket\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:52:36.114603 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.114403    2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-istio-podinfo\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:52:36.114603 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.114414    2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-credential-socket\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:52:36.114603 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.114424    2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-istio-token\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:52:36.114603 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.114432    2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-istio-envoy\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:52:36.114603 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.114440    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twqlx\" (UniqueName: \"kubernetes.io/projected/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b-kube-api-access-twqlx\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:52:36.962014 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:52:36.961980    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b" path="/var/lib/kubelet/pods/cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b/volumes"
Sep 09 15:53:35.267833 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:53:35.267799    2553 scope.go:117] "RemoveContainer" containerID="6ce0a72cf0d8710448c4d876eeb2ea8a1f78a3e2e5347d7fb24bae36e4b0afc8"
Sep 09 15:53:48.410190 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:53:48.410150    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-450a51/auth-test-model-450a51-kserve-86d7f44dcf-tlltv"]
Sep 09 15:53:50.029373 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:53:50.029341    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-482dce/auth-test-model-482dce-kserve-67d5796c5f-zpknh"]
Sep 09 15:53:58.605384 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:53:58.605350    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1e15c8-c8p45"]
Sep 09 15:53:58.605810 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:53:58.605741    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a4fbcf2-bbb3-471c-b11f-1673dd438efa" containerName="revoke-keys"
Sep 09 15:53:58.605810 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:53:58.605753    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a4fbcf2-bbb3-471c-b11f-1673dd438efa" containerName="revoke-keys"
Sep 09 15:53:58.605810 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:53:58.605767    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b" containerName="istio-proxy"
Sep 09 15:53:58.605810 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:53:58.605772    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b" containerName="istio-proxy"
Sep 09 15:53:58.605810 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:53:58.605786    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b" containerName="istio-proxy"
Sep 09 15:53:58.605810 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:53:58.605791    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b" containerName="istio-proxy"
Sep 09 15:53:58.605810 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:53:58.605798    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1eca92b-d2aa-4f69-85df-4a67708a08a9" containerName="payload-processing"
Sep 09 15:53:58.605810 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:53:58.605803    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1eca92b-d2aa-4f69-85df-4a67708a08a9" containerName="payload-processing"
Sep 09 15:53:58.606101 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:53:58.605872    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b" containerName="istio-proxy"
Sep 09 15:53:58.606101 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:53:58.605880    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1eca92b-d2aa-4f69-85df-4a67708a08a9" containerName="payload-processing"
Sep 09 15:53:58.606101 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:53:58.605886    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd3a5c3d-2ed5-4f09-bddd-87fafe3c715b" containerName="istio-proxy"
Sep 09 15:53:58.606101 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:53:58.605892    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="79946caa-e46d-4320-a5ec-80ad8a2c7bd9" containerName="istio-proxy"
Sep 09 15:53:58.606101 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:53:58.605900    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a4fbcf2-bbb3-471c-b11f-1673dd438efa" containerName="revoke-keys"
Sep 09 15:53:58.608328 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:53:58.608313    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1e15c8-c8p45"
Sep 09 15:53:58.611154 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:53:58.611130    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 15:53:58.611275 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:53:58.611130    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2zlsf\""
Sep 09 15:53:58.618421 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:53:58.618392    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1e15c8-c8p45"]
Sep 09 15:53:58.636825 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:53:58.636795    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c314dcd2-cca2-475e-8f5c-fdd62b477a98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-1e15c8-c8p45\" (UID: \"c314dcd2-cca2-475e-8f5c-fdd62b477a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1e15c8-c8p45"
Sep 09 15:53:58.737352 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:53:58.737314    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c314dcd2-cca2-475e-8f5c-fdd62b477a98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-1e15c8-c8p45\" (UID: \"c314dcd2-cca2-475e-8f5c-fdd62b477a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1e15c8-c8p45"
Sep 09 15:53:58.738186 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:53:58.738158    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c314dcd2-cca2-475e-8f5c-fdd62b477a98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-1e15c8-c8p45\" (UID: \"c314dcd2-cca2-475e-8f5c-fdd62b477a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1e15c8-c8p45"
Sep 09 15:53:58.919676 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:53:58.919642    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1e15c8-c8p45"
Sep 09 15:53:59.192006 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:53:59.191922    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1e15c8-c8p45"]
Sep 09 15:53:59.194374 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:53:59.194347    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc314dcd2_cca2_475e_8f5c_fdd62b477a98.slice/crio-6bf352e312b25656704d00f49fae9d6f9f680ffd1c4881bac7669c2a6547abea WatchSource:0}: Error finding container 6bf352e312b25656704d00f49fae9d6f9f680ffd1c4881bac7669c2a6547abea: Status 404 returned error can't find the container with id 6bf352e312b25656704d00f49fae9d6f9f680ffd1c4881bac7669c2a6547abea
Sep 09 15:53:59.493753 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:53:59.493698    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1e15c8-c8p45" event={"ID":"c314dcd2-cca2-475e-8f5c-fdd62b477a98","Type":"ContainerStarted","Data":"103fc768c2e9dd9d5067b2da6bef1b6b789c5856d53d2aff1a03c334ef336011"}
Sep 09 15:53:59.493914 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:53:59.493778    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1e15c8-c8p45" event={"ID":"c314dcd2-cca2-475e-8f5c-fdd62b477a98","Type":"ContainerStarted","Data":"6bf352e312b25656704d00f49fae9d6f9f680ffd1c4881bac7669c2a6547abea"}
Sep 09 15:53:59.509538 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:53:59.509486    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1e15c8-c8p45" podStartSLOduration=1.509470608 podStartE2EDuration="1.509470608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:53:59.507766476 +0000 UTC m=+2005.106661121" watchObservedRunningTime="2026-09-09 15:53:59.509470608 +0000 UTC m=+2005.108365252"
Sep 09 15:54:00.500613 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:00.500574    2553 generic.go:356] "Generic (PLEG): container finished" podID="c314dcd2-cca2-475e-8f5c-fdd62b477a98" containerID="103fc768c2e9dd9d5067b2da6bef1b6b789c5856d53d2aff1a03c334ef336011" exitCode=0
Sep 09 15:54:00.501020 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:00.500659    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1e15c8-c8p45" event={"ID":"c314dcd2-cca2-475e-8f5c-fdd62b477a98","Type":"ContainerDied","Data":"103fc768c2e9dd9d5067b2da6bef1b6b789c5856d53d2aff1a03c334ef336011"}
Sep 09 15:54:01.636875 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:01.636850    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1e15c8-c8p45"
Sep 09 15:54:01.667608 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:01.667579    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c314dcd2-cca2-475e-8f5c-fdd62b477a98-maas-api-service-ca\") pod \"c314dcd2-cca2-475e-8f5c-fdd62b477a98\" (UID: \"c314dcd2-cca2-475e-8f5c-fdd62b477a98\") "
Sep 09 15:54:01.667990 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:01.667964    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c314dcd2-cca2-475e-8f5c-fdd62b477a98-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c314dcd2-cca2-475e-8f5c-fdd62b477a98" (UID: "c314dcd2-cca2-475e-8f5c-fdd62b477a98"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:54:01.769020 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:01.768934    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c314dcd2-cca2-475e-8f5c-fdd62b477a98-maas-api-service-ca\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:02.512299 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:02.512268    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1e15c8-c8p45"
Sep 09 15:54:02.512502 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:02.512307    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1e15c8-c8p45" event={"ID":"c314dcd2-cca2-475e-8f5c-fdd62b477a98","Type":"ContainerDied","Data":"6bf352e312b25656704d00f49fae9d6f9f680ffd1c4881bac7669c2a6547abea"}
Sep 09 15:54:02.512502 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:02.512359    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bf352e312b25656704d00f49fae9d6f9f680ffd1c4881bac7669c2a6547abea"
Sep 09 15:54:03.430458 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:03.430416    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-450a51/auth-test-model-450a51-kserve-86d7f44dcf-tlltv" podUID="c70ce445-aedc-4429-8beb-0f21a23c357b" containerName="main" containerID="cri-o://8104f36df9ad1d14591d04cc9fc32e02d054e63419a09bf43b263226acea4109" gracePeriod=45
Sep 09 15:54:03.466698 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:03.466662    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg"]
Sep 09 15:54:03.499773 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:03.499723    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2"]
Sep 09 15:54:03.506690 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:03.506660    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86"]
Sep 09 15:54:03.681938 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:03.681880    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-450a51/auth-test-model-450a51-kserve-86d7f44dcf-tlltv"
Sep 09 15:54:03.790330 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:03.790292    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c70ce445-aedc-4429-8beb-0f21a23c357b-home\") pod \"c70ce445-aedc-4429-8beb-0f21a23c357b\" (UID: \"c70ce445-aedc-4429-8beb-0f21a23c357b\") "
Sep 09 15:54:03.790330 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:03.790338    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c70ce445-aedc-4429-8beb-0f21a23c357b-tls-certs\") pod \"c70ce445-aedc-4429-8beb-0f21a23c357b\" (UID: \"c70ce445-aedc-4429-8beb-0f21a23c357b\") "
Sep 09 15:54:03.790596 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:03.790362    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7s4jq\" (UniqueName: \"kubernetes.io/projected/c70ce445-aedc-4429-8beb-0f21a23c357b-kube-api-access-7s4jq\") pod \"c70ce445-aedc-4429-8beb-0f21a23c357b\" (UID: \"c70ce445-aedc-4429-8beb-0f21a23c357b\") "
Sep 09 15:54:03.790596 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:03.790394    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c70ce445-aedc-4429-8beb-0f21a23c357b-dshm\") pod \"c70ce445-aedc-4429-8beb-0f21a23c357b\" (UID: \"c70ce445-aedc-4429-8beb-0f21a23c357b\") "
Sep 09 15:54:03.790596 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:03.790484    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c70ce445-aedc-4429-8beb-0f21a23c357b-model-cache\") pod \"c70ce445-aedc-4429-8beb-0f21a23c357b\" (UID: \"c70ce445-aedc-4429-8beb-0f21a23c357b\") "
Sep 09 15:54:03.790596 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:03.790513    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c70ce445-aedc-4429-8beb-0f21a23c357b-tmp-dir\") pod \"c70ce445-aedc-4429-8beb-0f21a23c357b\" (UID: \"c70ce445-aedc-4429-8beb-0f21a23c357b\") "
Sep 09 15:54:03.790850 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:03.790643    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70ce445-aedc-4429-8beb-0f21a23c357b-home" (OuterVolumeSpecName: "home") pod "c70ce445-aedc-4429-8beb-0f21a23c357b" (UID: "c70ce445-aedc-4429-8beb-0f21a23c357b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:54:03.790850 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:03.790707    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70ce445-aedc-4429-8beb-0f21a23c357b-model-cache" (OuterVolumeSpecName: "model-cache") pod "c70ce445-aedc-4429-8beb-0f21a23c357b" (UID: "c70ce445-aedc-4429-8beb-0f21a23c357b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:54:03.790955 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:03.790897    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70ce445-aedc-4429-8beb-0f21a23c357b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c70ce445-aedc-4429-8beb-0f21a23c357b" (UID: "c70ce445-aedc-4429-8beb-0f21a23c357b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:54:03.790955 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:03.790921    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c70ce445-aedc-4429-8beb-0f21a23c357b-model-cache\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:03.790955 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:03.790936    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c70ce445-aedc-4429-8beb-0f21a23c357b-home\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:03.792773 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:03.792746    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c70ce445-aedc-4429-8beb-0f21a23c357b-kube-api-access-7s4jq" (OuterVolumeSpecName: "kube-api-access-7s4jq") pod "c70ce445-aedc-4429-8beb-0f21a23c357b" (UID: "c70ce445-aedc-4429-8beb-0f21a23c357b"). InnerVolumeSpecName "kube-api-access-7s4jq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:54:03.793011 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:03.792992    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70ce445-aedc-4429-8beb-0f21a23c357b-dshm" (OuterVolumeSpecName: "dshm") pod "c70ce445-aedc-4429-8beb-0f21a23c357b" (UID: "c70ce445-aedc-4429-8beb-0f21a23c357b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:54:03.793079 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:03.793005    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c70ce445-aedc-4429-8beb-0f21a23c357b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c70ce445-aedc-4429-8beb-0f21a23c357b" (UID: "c70ce445-aedc-4429-8beb-0f21a23c357b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:54:03.891712 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:03.891672    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c70ce445-aedc-4429-8beb-0f21a23c357b-tls-certs\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:03.891712 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:03.891711    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s4jq\" (UniqueName: \"kubernetes.io/projected/c70ce445-aedc-4429-8beb-0f21a23c357b-kube-api-access-7s4jq\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:03.892003 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:03.891724    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c70ce445-aedc-4429-8beb-0f21a23c357b-dshm\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:03.892003 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:03.891839    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c70ce445-aedc-4429-8beb-0f21a23c357b-tmp-dir\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:04.524357 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:04.524319    2553 generic.go:356] "Generic (PLEG): container finished" podID="c70ce445-aedc-4429-8beb-0f21a23c357b" containerID="8104f36df9ad1d14591d04cc9fc32e02d054e63419a09bf43b263226acea4109" exitCode=0
Sep 09 15:54:04.524772 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:04.524376    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-450a51/auth-test-model-450a51-kserve-86d7f44dcf-tlltv"
Sep 09 15:54:04.524772 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:04.524389    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-450a51/auth-test-model-450a51-kserve-86d7f44dcf-tlltv" event={"ID":"c70ce445-aedc-4429-8beb-0f21a23c357b","Type":"ContainerDied","Data":"8104f36df9ad1d14591d04cc9fc32e02d054e63419a09bf43b263226acea4109"}
Sep 09 15:54:04.524772 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:04.524424    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-450a51/auth-test-model-450a51-kserve-86d7f44dcf-tlltv" event={"ID":"c70ce445-aedc-4429-8beb-0f21a23c357b","Type":"ContainerDied","Data":"20f9d94ed77727c0629939843e4ffb87557eb8fd84f852b67d02ac7416569f06"}
Sep 09 15:54:04.524772 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:04.524442    2553 scope.go:117] "RemoveContainer" containerID="8104f36df9ad1d14591d04cc9fc32e02d054e63419a09bf43b263226acea4109"
Sep 09 15:54:04.534592 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:04.534572    2553 scope.go:117] "RemoveContainer" containerID="8104f36df9ad1d14591d04cc9fc32e02d054e63419a09bf43b263226acea4109"
Sep 09 15:54:04.534928 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:54:04.534908    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8104f36df9ad1d14591d04cc9fc32e02d054e63419a09bf43b263226acea4109\": container with ID starting with 8104f36df9ad1d14591d04cc9fc32e02d054e63419a09bf43b263226acea4109 not found: ID does not exist" containerID="8104f36df9ad1d14591d04cc9fc32e02d054e63419a09bf43b263226acea4109"
Sep 09 15:54:04.534989 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:04.534936    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8104f36df9ad1d14591d04cc9fc32e02d054e63419a09bf43b263226acea4109"} err="failed to get container status \"8104f36df9ad1d14591d04cc9fc32e02d054e63419a09bf43b263226acea4109\": rpc error: code = NotFound desc = could not find container \"8104f36df9ad1d14591d04cc9fc32e02d054e63419a09bf43b263226acea4109\": container with ID starting with 8104f36df9ad1d14591d04cc9fc32e02d054e63419a09bf43b263226acea4109 not found: ID does not exist"
Sep 09 15:54:04.545954 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:04.545920    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-450a51/auth-test-model-450a51-kserve-86d7f44dcf-tlltv"]
Sep 09 15:54:04.550380 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:04.550355    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-450a51/auth-test-model-450a51-kserve-86d7f44dcf-tlltv"]
Sep 09 15:54:04.962526 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:04.962484    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c70ce445-aedc-4429-8beb-0f21a23c357b" path="/var/lib/kubelet/pods/c70ce445-aedc-4429-8beb-0f21a23c357b/volumes"
Sep 09 15:54:05.054583 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:05.054542    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-482dce/auth-test-model-482dce-kserve-67d5796c5f-zpknh" podUID="cb2a60ce-3718-4c59-8c72-7608c97b6027" containerName="main" containerID="cri-o://14cfe5f010c9398abbcb4404dac529a86744e519639aae22a75d4b65235601f0" gracePeriod=45
Sep 09 15:54:05.303776 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:05.303750    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-482dce/auth-test-model-482dce-kserve-67d5796c5f-zpknh"
Sep 09 15:54:05.406802 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:05.406766    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb2a60ce-3718-4c59-8c72-7608c97b6027-tls-certs\") pod \"cb2a60ce-3718-4c59-8c72-7608c97b6027\" (UID: \"cb2a60ce-3718-4c59-8c72-7608c97b6027\") "
Sep 09 15:54:05.407006 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:05.406820    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb2a60ce-3718-4c59-8c72-7608c97b6027-dshm\") pod \"cb2a60ce-3718-4c59-8c72-7608c97b6027\" (UID: \"cb2a60ce-3718-4c59-8c72-7608c97b6027\") "
Sep 09 15:54:05.407006 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:05.406851    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sb6r\" (UniqueName: \"kubernetes.io/projected/cb2a60ce-3718-4c59-8c72-7608c97b6027-kube-api-access-2sb6r\") pod \"cb2a60ce-3718-4c59-8c72-7608c97b6027\" (UID: \"cb2a60ce-3718-4c59-8c72-7608c97b6027\") "
Sep 09 15:54:05.407006 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:05.406903    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb2a60ce-3718-4c59-8c72-7608c97b6027-tmp-dir\") pod \"cb2a60ce-3718-4c59-8c72-7608c97b6027\" (UID: \"cb2a60ce-3718-4c59-8c72-7608c97b6027\") "
Sep 09 15:54:05.407006 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:05.406934    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb2a60ce-3718-4c59-8c72-7608c97b6027-home\") pod \"cb2a60ce-3718-4c59-8c72-7608c97b6027\" (UID: \"cb2a60ce-3718-4c59-8c72-7608c97b6027\") "
Sep 09 15:54:05.407006 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:05.406950    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb2a60ce-3718-4c59-8c72-7608c97b6027-model-cache\") pod \"cb2a60ce-3718-4c59-8c72-7608c97b6027\" (UID: \"cb2a60ce-3718-4c59-8c72-7608c97b6027\") "
Sep 09 15:54:05.407305 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:05.407207    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb2a60ce-3718-4c59-8c72-7608c97b6027-home" (OuterVolumeSpecName: "home") pod "cb2a60ce-3718-4c59-8c72-7608c97b6027" (UID: "cb2a60ce-3718-4c59-8c72-7608c97b6027"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:54:05.407305 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:05.407218    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb2a60ce-3718-4c59-8c72-7608c97b6027-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cb2a60ce-3718-4c59-8c72-7608c97b6027" (UID: "cb2a60ce-3718-4c59-8c72-7608c97b6027"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:54:05.407417 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:05.407340    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb2a60ce-3718-4c59-8c72-7608c97b6027-model-cache" (OuterVolumeSpecName: "model-cache") pod "cb2a60ce-3718-4c59-8c72-7608c97b6027" (UID: "cb2a60ce-3718-4c59-8c72-7608c97b6027"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:54:05.407417 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:05.407369    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb2a60ce-3718-4c59-8c72-7608c97b6027-home\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:05.407417 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:05.407410    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb2a60ce-3718-4c59-8c72-7608c97b6027-tmp-dir\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:05.409190 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:05.409162    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb2a60ce-3718-4c59-8c72-7608c97b6027-dshm" (OuterVolumeSpecName: "dshm") pod "cb2a60ce-3718-4c59-8c72-7608c97b6027" (UID: "cb2a60ce-3718-4c59-8c72-7608c97b6027"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:54:05.409294 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:05.409191    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb2a60ce-3718-4c59-8c72-7608c97b6027-kube-api-access-2sb6r" (OuterVolumeSpecName: "kube-api-access-2sb6r") pod "cb2a60ce-3718-4c59-8c72-7608c97b6027" (UID: "cb2a60ce-3718-4c59-8c72-7608c97b6027"). InnerVolumeSpecName "kube-api-access-2sb6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:54:05.409294 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:05.409212    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb2a60ce-3718-4c59-8c72-7608c97b6027-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cb2a60ce-3718-4c59-8c72-7608c97b6027" (UID: "cb2a60ce-3718-4c59-8c72-7608c97b6027"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:54:05.509045 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:05.508944    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb2a60ce-3718-4c59-8c72-7608c97b6027-model-cache\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:05.509045 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:05.508980    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb2a60ce-3718-4c59-8c72-7608c97b6027-tls-certs\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:05.509045 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:05.508991    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb2a60ce-3718-4c59-8c72-7608c97b6027-dshm\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:05.509045 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:05.509003    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sb6r\" (UniqueName: \"kubernetes.io/projected/cb2a60ce-3718-4c59-8c72-7608c97b6027-kube-api-access-2sb6r\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:05.530640 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:05.530609    2553 generic.go:356] "Generic (PLEG): container finished" podID="cb2a60ce-3718-4c59-8c72-7608c97b6027" containerID="14cfe5f010c9398abbcb4404dac529a86744e519639aae22a75d4b65235601f0" exitCode=0
Sep 09 15:54:05.531141 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:05.530663    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-482dce/auth-test-model-482dce-kserve-67d5796c5f-zpknh"
Sep 09 15:54:05.531141 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:05.530700    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-482dce/auth-test-model-482dce-kserve-67d5796c5f-zpknh" event={"ID":"cb2a60ce-3718-4c59-8c72-7608c97b6027","Type":"ContainerDied","Data":"14cfe5f010c9398abbcb4404dac529a86744e519639aae22a75d4b65235601f0"}
Sep 09 15:54:05.531141 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:05.530771    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-482dce/auth-test-model-482dce-kserve-67d5796c5f-zpknh" event={"ID":"cb2a60ce-3718-4c59-8c72-7608c97b6027","Type":"ContainerDied","Data":"dc61e30a7803e5f79481c6d3700aa94191e36c5c37e3a64d48468da86b2065b7"}
Sep 09 15:54:05.531141 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:05.530792    2553 scope.go:117] "RemoveContainer" containerID="14cfe5f010c9398abbcb4404dac529a86744e519639aae22a75d4b65235601f0"
Sep 09 15:54:05.540493 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:05.540473    2553 scope.go:117] "RemoveContainer" containerID="14cfe5f010c9398abbcb4404dac529a86744e519639aae22a75d4b65235601f0"
Sep 09 15:54:05.540840 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:54:05.540816    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14cfe5f010c9398abbcb4404dac529a86744e519639aae22a75d4b65235601f0\": container with ID starting with 14cfe5f010c9398abbcb4404dac529a86744e519639aae22a75d4b65235601f0 not found: ID does not exist" containerID="14cfe5f010c9398abbcb4404dac529a86744e519639aae22a75d4b65235601f0"
Sep 09 15:54:05.540962 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:05.540847    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14cfe5f010c9398abbcb4404dac529a86744e519639aae22a75d4b65235601f0"} err="failed to get container status \"14cfe5f010c9398abbcb4404dac529a86744e519639aae22a75d4b65235601f0\": rpc error: code = NotFound desc = could not find container \"14cfe5f010c9398abbcb4404dac529a86744e519639aae22a75d4b65235601f0\": container with ID starting with 14cfe5f010c9398abbcb4404dac529a86744e519639aae22a75d4b65235601f0 not found: ID does not exist"
Sep 09 15:54:05.551657 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:05.551628    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-482dce/auth-test-model-482dce-kserve-67d5796c5f-zpknh"]
Sep 09 15:54:05.553547 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:05.553527    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-482dce/auth-test-model-482dce-kserve-67d5796c5f-zpknh"]
Sep 09 15:54:06.963981 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:06.963939    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb2a60ce-3718-4c59-8c72-7608c97b6027" path="/var/lib/kubelet/pods/cb2a60ce-3718-4c59-8c72-7608c97b6027/volumes"
Sep 09 15:54:14.277120 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:14.277086    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1e15c8-29816145-kzt6s"]
Sep 09 15:54:14.282265 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:14.282236    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1e15c8-29816145-kzt6s"]
Sep 09 15:54:14.310675 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:14.310639    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1e15c8-7c96479557-8f8jr"]
Sep 09 15:54:14.311011 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:14.310972    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-1e15c8-7c96479557-8f8jr" podUID="39cb9b9a-9e10-4879-ad05-99155842ca9e" containerName="payload-processing" containerID="cri-o://13f1abea79589731d7be57557aac915377b3ce68926e705d0583d2bbb049fe6b" gracePeriod=30
Sep 09 15:54:14.318513 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:14.318488    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-1e15c8-6f968d8478-8gkmw"]
Sep 09 15:54:14.319015 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:14.318779    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1e15c8-6f968d8478-8gkmw" podUID="1152d276-323d-4e1a-b67e-a2ce9927ae59" containerName="payload-pre-processing" containerID="cri-o://53f4cec957b58b7e641fb97ed0a3ea04a86addd43dbdd026088bd4ccca26ea75" gracePeriod=30
Sep 09 15:54:14.487777 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:14.487718    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1e15c8-c8p45"]
Sep 09 15:54:14.491988 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:14.491959    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1e15c8-c8p45"]
Sep 09 15:54:14.587634 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:14.587606    2553 generic.go:356] "Generic (PLEG): container finished" podID="1152d276-323d-4e1a-b67e-a2ce9927ae59" containerID="53f4cec957b58b7e641fb97ed0a3ea04a86addd43dbdd026088bd4ccca26ea75" exitCode=0
Sep 09 15:54:14.587779 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:14.587686    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1e15c8-6f968d8478-8gkmw" event={"ID":"1152d276-323d-4e1a-b67e-a2ce9927ae59","Type":"ContainerDied","Data":"53f4cec957b58b7e641fb97ed0a3ea04a86addd43dbdd026088bd4ccca26ea75"}
Sep 09 15:54:14.587858 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:14.587785    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1e15c8-6f968d8478-8gkmw" event={"ID":"1152d276-323d-4e1a-b67e-a2ce9927ae59","Type":"ContainerDied","Data":"8c602ed72f1505cac152aab82386f954840021f61ee7e8ecefb7275404f8b74c"}
Sep 09 15:54:14.587858 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:14.587816    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c602ed72f1505cac152aab82386f954840021f61ee7e8ecefb7275404f8b74c"
Sep 09 15:54:14.589141 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:14.589122    2553 generic.go:356] "Generic (PLEG): container finished" podID="39cb9b9a-9e10-4879-ad05-99155842ca9e" containerID="13f1abea79589731d7be57557aac915377b3ce68926e705d0583d2bbb049fe6b" exitCode=0
Sep 09 15:54:14.589233 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:14.589166    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-1e15c8-7c96479557-8f8jr" event={"ID":"39cb9b9a-9e10-4879-ad05-99155842ca9e","Type":"ContainerDied","Data":"13f1abea79589731d7be57557aac915377b3ce68926e705d0583d2bbb049fe6b"}
Sep 09 15:54:14.589233 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:14.589205    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-1e15c8-7c96479557-8f8jr" event={"ID":"39cb9b9a-9e10-4879-ad05-99155842ca9e","Type":"ContainerDied","Data":"a64842e58c1b7f1b87edbe775adba4c26ccc6387d9c44f49b129c04a61e6877c"}
Sep 09 15:54:14.589233 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:14.589220    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a64842e58c1b7f1b87edbe775adba4c26ccc6387d9c44f49b129c04a61e6877c"
Sep 09 15:54:14.599210 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:14.599187    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-1e15c8-7c96479557-8f8jr"
Sep 09 15:54:14.602191 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:14.602173    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1e15c8-6f968d8478-8gkmw"
Sep 09 15:54:14.701238 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:14.701205    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1152d276-323d-4e1a-b67e-a2ce9927ae59-plugins-config-volume\") pod \"1152d276-323d-4e1a-b67e-a2ce9927ae59\" (UID: \"1152d276-323d-4e1a-b67e-a2ce9927ae59\") "
Sep 09 15:54:14.701238 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:14.701241    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mg6t\" (UniqueName: \"kubernetes.io/projected/1152d276-323d-4e1a-b67e-a2ce9927ae59-kube-api-access-8mg6t\") pod \"1152d276-323d-4e1a-b67e-a2ce9927ae59\" (UID: \"1152d276-323d-4e1a-b67e-a2ce9927ae59\") "
Sep 09 15:54:14.701485 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:14.701283    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cmf4\" (UniqueName: \"kubernetes.io/projected/39cb9b9a-9e10-4879-ad05-99155842ca9e-kube-api-access-9cmf4\") pod \"39cb9b9a-9e10-4879-ad05-99155842ca9e\" (UID: \"39cb9b9a-9e10-4879-ad05-99155842ca9e\") "
Sep 09 15:54:14.701485 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:14.701323    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39cb9b9a-9e10-4879-ad05-99155842ca9e-plugins-config-volume\") pod \"39cb9b9a-9e10-4879-ad05-99155842ca9e\" (UID: \"39cb9b9a-9e10-4879-ad05-99155842ca9e\") "
Sep 09 15:54:14.701587 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:14.701562    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1152d276-323d-4e1a-b67e-a2ce9927ae59-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1152d276-323d-4e1a-b67e-a2ce9927ae59" (UID: "1152d276-323d-4e1a-b67e-a2ce9927ae59"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:54:14.701799 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:14.701757    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39cb9b9a-9e10-4879-ad05-99155842ca9e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "39cb9b9a-9e10-4879-ad05-99155842ca9e" (UID: "39cb9b9a-9e10-4879-ad05-99155842ca9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:54:14.703476 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:14.703453    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39cb9b9a-9e10-4879-ad05-99155842ca9e-kube-api-access-9cmf4" (OuterVolumeSpecName: "kube-api-access-9cmf4") pod "39cb9b9a-9e10-4879-ad05-99155842ca9e" (UID: "39cb9b9a-9e10-4879-ad05-99155842ca9e"). InnerVolumeSpecName "kube-api-access-9cmf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:54:14.703551 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:14.703484    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1152d276-323d-4e1a-b67e-a2ce9927ae59-kube-api-access-8mg6t" (OuterVolumeSpecName: "kube-api-access-8mg6t") pod "1152d276-323d-4e1a-b67e-a2ce9927ae59" (UID: "1152d276-323d-4e1a-b67e-a2ce9927ae59"). InnerVolumeSpecName "kube-api-access-8mg6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:54:14.802903 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:14.802853    2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39cb9b9a-9e10-4879-ad05-99155842ca9e-plugins-config-volume\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:14.802903 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:14.802900    2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1152d276-323d-4e1a-b67e-a2ce9927ae59-plugins-config-volume\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:14.802903 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:14.802910    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mg6t\" (UniqueName: \"kubernetes.io/projected/1152d276-323d-4e1a-b67e-a2ce9927ae59-kube-api-access-8mg6t\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:14.803152 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:14.802919    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cmf4\" (UniqueName: \"kubernetes.io/projected/39cb9b9a-9e10-4879-ad05-99155842ca9e-kube-api-access-9cmf4\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:14.962486 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:14.962450    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99226359-e48c-455a-9c80-4e8c21dae0a4" path="/var/lib/kubelet/pods/99226359-e48c-455a-9c80-4e8c21dae0a4/volumes"
Sep 09 15:54:14.962785 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:14.962772    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c314dcd2-cca2-475e-8f5c-fdd62b477a98" path="/var/lib/kubelet/pods/c314dcd2-cca2-475e-8f5c-fdd62b477a98/volumes"
Sep 09 15:54:15.600127 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:15.600021    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-1e15c8-7c96479557-8f8jr"
Sep 09 15:54:15.600589 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:15.600028    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1e15c8-6f968d8478-8gkmw"
Sep 09 15:54:15.620969 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:15.620936    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-1e15c8-6f968d8478-8gkmw"]
Sep 09 15:54:15.625909 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:15.625882    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-1e15c8-6f968d8478-8gkmw"]
Sep 09 15:54:15.637304 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:15.637277    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1e15c8-7c96479557-8f8jr"]
Sep 09 15:54:15.641479 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:15.641455    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1e15c8-7c96479557-8f8jr"]
Sep 09 15:54:16.963667 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:16.963632    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1152d276-323d-4e1a-b67e-a2ce9927ae59" path="/var/lib/kubelet/pods/1152d276-323d-4e1a-b67e-a2ce9927ae59/volumes"
Sep 09 15:54:16.964037 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:16.964006    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39cb9b9a-9e10-4879-ad05-99155842ca9e" path="/var/lib/kubelet/pods/39cb9b9a-9e10-4879-ad05-99155842ca9e/volumes"
Sep 09 15:54:18.486806 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.486765    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-1e15c8/distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg" podUID="cd4ed05e-a093-4371-9e6b-6983225f91b5" containerName="main" containerID="cri-o://7f11e2aecaafa78df1f0787eee91fe5a6a9ca3630eb75b0ba47198ee2b208231" gracePeriod=45
Sep 09 15:54:18.520743 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.520687    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-1e15c8/simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2" podUID="cc074101-6a71-45f8-8ba8-24a224e135bd" containerName="main" containerID="cri-o://fcd54b86b765bea721385a0cbb5c955c725661d2077f15b8ff4240b09c9a649f" gracePeriod=45
Sep 09 15:54:18.527719 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.527684    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-1e15c8/unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86" podUID="7eeea333-89f5-4c49-bb04-6b860ef79ad8" containerName="main" containerID="cri-o://603fd235c570cca63eeaeef18ad554c448bf740c4a399f0a382bbf1534c6fbb2" gracePeriod=45
Sep 09 15:54:18.619246 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.619219    2553 generic.go:356] "Generic (PLEG): container finished" podID="cd4ed05e-a093-4371-9e6b-6983225f91b5" containerID="7f11e2aecaafa78df1f0787eee91fe5a6a9ca3630eb75b0ba47198ee2b208231" exitCode=0
Sep 09 15:54:18.619386 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.619300    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1e15c8/distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg" event={"ID":"cd4ed05e-a093-4371-9e6b-6983225f91b5","Type":"ContainerDied","Data":"7f11e2aecaafa78df1f0787eee91fe5a6a9ca3630eb75b0ba47198ee2b208231"}
Sep 09 15:54:18.849187 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.849160    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1e15c8/distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg"
Sep 09 15:54:18.851166 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.851149    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1e15c8/simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2"
Sep 09 15:54:18.853309 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.853294    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1e15c8/unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86"
Sep 09 15:54:18.939879 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.939845    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7eeea333-89f5-4c49-bb04-6b860ef79ad8-home\") pod \"7eeea333-89f5-4c49-bb04-6b860ef79ad8\" (UID: \"7eeea333-89f5-4c49-bb04-6b860ef79ad8\") "
Sep 09 15:54:18.940064 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.939898    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7eeea333-89f5-4c49-bb04-6b860ef79ad8-dshm\") pod \"7eeea333-89f5-4c49-bb04-6b860ef79ad8\" (UID: \"7eeea333-89f5-4c49-bb04-6b860ef79ad8\") "
Sep 09 15:54:18.940064 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.939938    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmvkh\" (UniqueName: \"kubernetes.io/projected/cc074101-6a71-45f8-8ba8-24a224e135bd-kube-api-access-pmvkh\") pod \"cc074101-6a71-45f8-8ba8-24a224e135bd\" (UID: \"cc074101-6a71-45f8-8ba8-24a224e135bd\") "
Sep 09 15:54:18.940064 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.939965    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4ed05e-a093-4371-9e6b-6983225f91b5-tls-certs\") pod \"cd4ed05e-a093-4371-9e6b-6983225f91b5\" (UID: \"cd4ed05e-a093-4371-9e6b-6983225f91b5\") "
Sep 09 15:54:18.940064 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.939996    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd4ed05e-a093-4371-9e6b-6983225f91b5-dshm\") pod \"cd4ed05e-a093-4371-9e6b-6983225f91b5\" (UID: \"cd4ed05e-a093-4371-9e6b-6983225f91b5\") "
Sep 09 15:54:18.940064 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.940022    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7eeea333-89f5-4c49-bb04-6b860ef79ad8-model-cache\") pod \"7eeea333-89f5-4c49-bb04-6b860ef79ad8\" (UID: \"7eeea333-89f5-4c49-bb04-6b860ef79ad8\") "
Sep 09 15:54:18.940064 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.940049    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd4ed05e-a093-4371-9e6b-6983225f91b5-model-cache\") pod \"cd4ed05e-a093-4371-9e6b-6983225f91b5\" (UID: \"cd4ed05e-a093-4371-9e6b-6983225f91b5\") "
Sep 09 15:54:18.940379 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.940076    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd4ed05e-a093-4371-9e6b-6983225f91b5-home\") pod \"cd4ed05e-a093-4371-9e6b-6983225f91b5\" (UID: \"cd4ed05e-a093-4371-9e6b-6983225f91b5\") "
Sep 09 15:54:18.940379 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.940110    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92v7b\" (UniqueName: \"kubernetes.io/projected/cd4ed05e-a093-4371-9e6b-6983225f91b5-kube-api-access-92v7b\") pod \"cd4ed05e-a093-4371-9e6b-6983225f91b5\" (UID: \"cd4ed05e-a093-4371-9e6b-6983225f91b5\") "
Sep 09 15:54:18.940379 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.940141    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc074101-6a71-45f8-8ba8-24a224e135bd-tmp-dir\") pod \"cc074101-6a71-45f8-8ba8-24a224e135bd\" (UID: \"cc074101-6a71-45f8-8ba8-24a224e135bd\") "
Sep 09 15:54:18.940379 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.940162    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fsz8\" (UniqueName: \"kubernetes.io/projected/7eeea333-89f5-4c49-bb04-6b860ef79ad8-kube-api-access-2fsz8\") pod \"7eeea333-89f5-4c49-bb04-6b860ef79ad8\" (UID: \"7eeea333-89f5-4c49-bb04-6b860ef79ad8\") "
Sep 09 15:54:18.940379 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.940163    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eeea333-89f5-4c49-bb04-6b860ef79ad8-home" (OuterVolumeSpecName: "home") pod "7eeea333-89f5-4c49-bb04-6b860ef79ad8" (UID: "7eeea333-89f5-4c49-bb04-6b860ef79ad8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:54:18.940379 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.940205    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc074101-6a71-45f8-8ba8-24a224e135bd-home\") pod \"cc074101-6a71-45f8-8ba8-24a224e135bd\" (UID: \"cc074101-6a71-45f8-8ba8-24a224e135bd\") "
Sep 09 15:54:18.940379 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.940229    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc074101-6a71-45f8-8ba8-24a224e135bd-tls-certs\") pod \"cc074101-6a71-45f8-8ba8-24a224e135bd\" (UID: \"cc074101-6a71-45f8-8ba8-24a224e135bd\") "
Sep 09 15:54:18.940379 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.940251    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7eeea333-89f5-4c49-bb04-6b860ef79ad8-tls-certs\") pod \"7eeea333-89f5-4c49-bb04-6b860ef79ad8\" (UID: \"7eeea333-89f5-4c49-bb04-6b860ef79ad8\") "
Sep 09 15:54:18.940379 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.940279    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc074101-6a71-45f8-8ba8-24a224e135bd-model-cache\") pod \"cc074101-6a71-45f8-8ba8-24a224e135bd\" (UID: \"cc074101-6a71-45f8-8ba8-24a224e135bd\") "
Sep 09 15:54:18.940379 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.940301    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc074101-6a71-45f8-8ba8-24a224e135bd-dshm\") pod \"cc074101-6a71-45f8-8ba8-24a224e135bd\" (UID: \"cc074101-6a71-45f8-8ba8-24a224e135bd\") "
Sep 09 15:54:18.940379 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.940325    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7eeea333-89f5-4c49-bb04-6b860ef79ad8-tmp-dir\") pod \"7eeea333-89f5-4c49-bb04-6b860ef79ad8\" (UID: \"7eeea333-89f5-4c49-bb04-6b860ef79ad8\") "
Sep 09 15:54:18.940379 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.940363    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd4ed05e-a093-4371-9e6b-6983225f91b5-tmp-dir\") pod \"cd4ed05e-a093-4371-9e6b-6983225f91b5\" (UID: \"cd4ed05e-a093-4371-9e6b-6983225f91b5\") "
Sep 09 15:54:18.940379 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.940372    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd4ed05e-a093-4371-9e6b-6983225f91b5-model-cache" (OuterVolumeSpecName: "model-cache") pod "cd4ed05e-a093-4371-9e6b-6983225f91b5" (UID: "cd4ed05e-a093-4371-9e6b-6983225f91b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:54:18.941089 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.940631    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7eeea333-89f5-4c49-bb04-6b860ef79ad8-home\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:18.941089 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.940645    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd4ed05e-a093-4371-9e6b-6983225f91b5-model-cache\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:18.941089 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.940898    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd4ed05e-a093-4371-9e6b-6983225f91b5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cd4ed05e-a093-4371-9e6b-6983225f91b5" (UID: "cd4ed05e-a093-4371-9e6b-6983225f91b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:54:18.941089 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.941070    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc074101-6a71-45f8-8ba8-24a224e135bd-home" (OuterVolumeSpecName: "home") pod "cc074101-6a71-45f8-8ba8-24a224e135bd" (UID: "cc074101-6a71-45f8-8ba8-24a224e135bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:54:18.942334 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.941995    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc074101-6a71-45f8-8ba8-24a224e135bd-model-cache" (OuterVolumeSpecName: "model-cache") pod "cc074101-6a71-45f8-8ba8-24a224e135bd" (UID: "cc074101-6a71-45f8-8ba8-24a224e135bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:54:18.942334 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.942124    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eeea333-89f5-4c49-bb04-6b860ef79ad8-model-cache" (OuterVolumeSpecName: "model-cache") pod "7eeea333-89f5-4c49-bb04-6b860ef79ad8" (UID: "7eeea333-89f5-4c49-bb04-6b860ef79ad8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:54:18.943089 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.943056    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd4ed05e-a093-4371-9e6b-6983225f91b5-home" (OuterVolumeSpecName: "home") pod "cd4ed05e-a093-4371-9e6b-6983225f91b5" (UID: "cd4ed05e-a093-4371-9e6b-6983225f91b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:54:18.943187 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.943134    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc074101-6a71-45f8-8ba8-24a224e135bd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cc074101-6a71-45f8-8ba8-24a224e135bd" (UID: "cc074101-6a71-45f8-8ba8-24a224e135bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:54:18.943393 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.943371    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eeea333-89f5-4c49-bb04-6b860ef79ad8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7eeea333-89f5-4c49-bb04-6b860ef79ad8" (UID: "7eeea333-89f5-4c49-bb04-6b860ef79ad8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:54:18.943477 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.943437    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd4ed05e-a093-4371-9e6b-6983225f91b5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cd4ed05e-a093-4371-9e6b-6983225f91b5" (UID: "cd4ed05e-a093-4371-9e6b-6983225f91b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:54:18.943670 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.943640    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc074101-6a71-45f8-8ba8-24a224e135bd-kube-api-access-pmvkh" (OuterVolumeSpecName: "kube-api-access-pmvkh") pod "cc074101-6a71-45f8-8ba8-24a224e135bd" (UID: "cc074101-6a71-45f8-8ba8-24a224e135bd"). InnerVolumeSpecName "kube-api-access-pmvkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:54:18.944568 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.944534    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eeea333-89f5-4c49-bb04-6b860ef79ad8-dshm" (OuterVolumeSpecName: "dshm") pod "7eeea333-89f5-4c49-bb04-6b860ef79ad8" (UID: "7eeea333-89f5-4c49-bb04-6b860ef79ad8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:54:18.944665 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.944580    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc074101-6a71-45f8-8ba8-24a224e135bd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cc074101-6a71-45f8-8ba8-24a224e135bd" (UID: "cc074101-6a71-45f8-8ba8-24a224e135bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:54:18.944998 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.944954    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eeea333-89f5-4c49-bb04-6b860ef79ad8-kube-api-access-2fsz8" (OuterVolumeSpecName: "kube-api-access-2fsz8") pod "7eeea333-89f5-4c49-bb04-6b860ef79ad8" (UID: "7eeea333-89f5-4c49-bb04-6b860ef79ad8"). InnerVolumeSpecName "kube-api-access-2fsz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:54:18.945322 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.945299    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd4ed05e-a093-4371-9e6b-6983225f91b5-dshm" (OuterVolumeSpecName: "dshm") pod "cd4ed05e-a093-4371-9e6b-6983225f91b5" (UID: "cd4ed05e-a093-4371-9e6b-6983225f91b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:54:18.945447 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.945429    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eeea333-89f5-4c49-bb04-6b860ef79ad8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7eeea333-89f5-4c49-bb04-6b860ef79ad8" (UID: "7eeea333-89f5-4c49-bb04-6b860ef79ad8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:54:18.945597 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.945580    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc074101-6a71-45f8-8ba8-24a224e135bd-dshm" (OuterVolumeSpecName: "dshm") pod "cc074101-6a71-45f8-8ba8-24a224e135bd" (UID: "cc074101-6a71-45f8-8ba8-24a224e135bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:54:18.945970 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:18.945952    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd4ed05e-a093-4371-9e6b-6983225f91b5-kube-api-access-92v7b" (OuterVolumeSpecName: "kube-api-access-92v7b") pod "cd4ed05e-a093-4371-9e6b-6983225f91b5" (UID: "cd4ed05e-a093-4371-9e6b-6983225f91b5"). InnerVolumeSpecName "kube-api-access-92v7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:54:19.041774 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:19.041642    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmvkh\" (UniqueName: \"kubernetes.io/projected/cc074101-6a71-45f8-8ba8-24a224e135bd-kube-api-access-pmvkh\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:19.041774 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:19.041683    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4ed05e-a093-4371-9e6b-6983225f91b5-tls-certs\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:19.041774 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:19.041700    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd4ed05e-a093-4371-9e6b-6983225f91b5-dshm\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:19.041774 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:19.041712    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7eeea333-89f5-4c49-bb04-6b860ef79ad8-model-cache\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:19.041774 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:19.041724    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd4ed05e-a093-4371-9e6b-6983225f91b5-home\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:19.041774 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:19.041753    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92v7b\" (UniqueName: \"kubernetes.io/projected/cd4ed05e-a093-4371-9e6b-6983225f91b5-kube-api-access-92v7b\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:19.041774 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:19.041765    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc074101-6a71-45f8-8ba8-24a224e135bd-tmp-dir\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:19.041774 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:19.041781    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fsz8\" (UniqueName: \"kubernetes.io/projected/7eeea333-89f5-4c49-bb04-6b860ef79ad8-kube-api-access-2fsz8\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:19.042354 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:19.041793    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc074101-6a71-45f8-8ba8-24a224e135bd-home\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:19.042354 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:19.041805    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc074101-6a71-45f8-8ba8-24a224e135bd-tls-certs\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:19.042354 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:19.041817    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7eeea333-89f5-4c49-bb04-6b860ef79ad8-tls-certs\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:19.042354 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:19.041829    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc074101-6a71-45f8-8ba8-24a224e135bd-model-cache\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:19.042354 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:19.041840    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc074101-6a71-45f8-8ba8-24a224e135bd-dshm\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:19.042354 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:19.041852    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7eeea333-89f5-4c49-bb04-6b860ef79ad8-tmp-dir\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:19.042354 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:19.041864    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd4ed05e-a093-4371-9e6b-6983225f91b5-tmp-dir\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:19.042354 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:19.041885    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7eeea333-89f5-4c49-bb04-6b860ef79ad8-dshm\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:19.626429 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:19.626389    2553 generic.go:356] "Generic (PLEG): container finished" podID="cc074101-6a71-45f8-8ba8-24a224e135bd" containerID="fcd54b86b765bea721385a0cbb5c955c725661d2077f15b8ff4240b09c9a649f" exitCode=0
Sep 09 15:54:19.626944 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:19.626432    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1e15c8/simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2"
Sep 09 15:54:19.626944 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:19.626475    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1e15c8/simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2" event={"ID":"cc074101-6a71-45f8-8ba8-24a224e135bd","Type":"ContainerDied","Data":"fcd54b86b765bea721385a0cbb5c955c725661d2077f15b8ff4240b09c9a649f"}
Sep 09 15:54:19.626944 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:19.626518    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1e15c8/simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2" event={"ID":"cc074101-6a71-45f8-8ba8-24a224e135bd","Type":"ContainerDied","Data":"ecc08c463ffb4f6f617c36186e19ac07462370fa173f95caef341dc91939dd03"}
Sep 09 15:54:19.626944 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:19.626541    2553 scope.go:117] "RemoveContainer" containerID="fcd54b86b765bea721385a0cbb5c955c725661d2077f15b8ff4240b09c9a649f"
Sep 09 15:54:19.628128 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:19.628072    2553 generic.go:356] "Generic (PLEG): container finished" podID="7eeea333-89f5-4c49-bb04-6b860ef79ad8" containerID="603fd235c570cca63eeaeef18ad554c448bf740c4a399f0a382bbf1534c6fbb2" exitCode=0
Sep 09 15:54:19.628128 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:19.628106    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1e15c8/unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86"
Sep 09 15:54:19.628250 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:19.628164    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1e15c8/unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86" event={"ID":"7eeea333-89f5-4c49-bb04-6b860ef79ad8","Type":"ContainerDied","Data":"603fd235c570cca63eeaeef18ad554c448bf740c4a399f0a382bbf1534c6fbb2"}
Sep 09 15:54:19.628250 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:19.628206    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1e15c8/unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86" event={"ID":"7eeea333-89f5-4c49-bb04-6b860ef79ad8","Type":"ContainerDied","Data":"2b35b6bd81dfe7510edd2bfdfcdb9359e6c887ca416b7bb5c2875b656b4efa60"}
Sep 09 15:54:19.629795 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:19.629777    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1e15c8/distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg"
Sep 09 15:54:19.629916 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:19.629777    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1e15c8/distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg" event={"ID":"cd4ed05e-a093-4371-9e6b-6983225f91b5","Type":"ContainerDied","Data":"c6ff0a5349eebf4b733bdfedfd5d9ef05f1d89787de306bbe6b5ab753985d197"}
Sep 09 15:54:19.636180 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:19.636163    2553 scope.go:117] "RemoveContainer" containerID="fcd54b86b765bea721385a0cbb5c955c725661d2077f15b8ff4240b09c9a649f"
Sep 09 15:54:19.636461 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:54:19.636443    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcd54b86b765bea721385a0cbb5c955c725661d2077f15b8ff4240b09c9a649f\": container with ID starting with fcd54b86b765bea721385a0cbb5c955c725661d2077f15b8ff4240b09c9a649f not found: ID does not exist" containerID="fcd54b86b765bea721385a0cbb5c955c725661d2077f15b8ff4240b09c9a649f"
Sep 09 15:54:19.636503 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:19.636473    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcd54b86b765bea721385a0cbb5c955c725661d2077f15b8ff4240b09c9a649f"} err="failed to get container status \"fcd54b86b765bea721385a0cbb5c955c725661d2077f15b8ff4240b09c9a649f\": rpc error: code = NotFound desc = could not find container \"fcd54b86b765bea721385a0cbb5c955c725661d2077f15b8ff4240b09c9a649f\": container with ID starting with fcd54b86b765bea721385a0cbb5c955c725661d2077f15b8ff4240b09c9a649f not found: ID does not exist"
Sep 09 15:54:19.636503 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:19.636489    2553 scope.go:117] "RemoveContainer" containerID="603fd235c570cca63eeaeef18ad554c448bf740c4a399f0a382bbf1534c6fbb2"
Sep 09 15:54:19.645252 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:19.645226    2553 scope.go:117] "RemoveContainer" containerID="603fd235c570cca63eeaeef18ad554c448bf740c4a399f0a382bbf1534c6fbb2"
Sep 09 15:54:19.645574 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:54:19.645551    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"603fd235c570cca63eeaeef18ad554c448bf740c4a399f0a382bbf1534c6fbb2\": container with ID starting with 603fd235c570cca63eeaeef18ad554c448bf740c4a399f0a382bbf1534c6fbb2 not found: ID does not exist" containerID="603fd235c570cca63eeaeef18ad554c448bf740c4a399f0a382bbf1534c6fbb2"
Sep 09 15:54:19.645708 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:19.645608    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"603fd235c570cca63eeaeef18ad554c448bf740c4a399f0a382bbf1534c6fbb2"} err="failed to get container status \"603fd235c570cca63eeaeef18ad554c448bf740c4a399f0a382bbf1534c6fbb2\": rpc error: code = NotFound desc = could not find container \"603fd235c570cca63eeaeef18ad554c448bf740c4a399f0a382bbf1534c6fbb2\": container with ID starting with 603fd235c570cca63eeaeef18ad554c448bf740c4a399f0a382bbf1534c6fbb2 not found: ID does not exist"
Sep 09 15:54:19.645708 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:19.645636    2553 scope.go:117] "RemoveContainer" containerID="7f11e2aecaafa78df1f0787eee91fe5a6a9ca3630eb75b0ba47198ee2b208231"
Sep 09 15:54:19.648497 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:19.648471    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2"]
Sep 09 15:54:19.651358 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:19.651335    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/simulator-w4-1e15c8-kserve-5b8d8fb45b-nh5d2"]
Sep 09 15:54:19.665832 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:19.665803    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86"]
Sep 09 15:54:19.675431 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:19.672820    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/unconfigured-w4-1e15c8-kserve-7cfddf6559-g5h86"]
Sep 09 15:54:19.684450 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:19.684424    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg"]
Sep 09 15:54:19.687141 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:19.687114    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/distinct-2-w4-1e15c8-kserve-5889df595f-kxnxg"]
Sep 09 15:54:20.963710 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:20.963672    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7eeea333-89f5-4c49-bb04-6b860ef79ad8" path="/var/lib/kubelet/pods/7eeea333-89f5-4c49-bb04-6b860ef79ad8/volumes"
Sep 09 15:54:20.964106 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:20.964090    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc074101-6a71-45f8-8ba8-24a224e135bd" path="/var/lib/kubelet/pods/cc074101-6a71-45f8-8ba8-24a224e135bd/volumes"
Sep 09 15:54:20.964451 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:20.964438    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd4ed05e-a093-4371-9e6b-6983225f91b5" path="/var/lib/kubelet/pods/cd4ed05e-a093-4371-9e6b-6983225f91b5/volumes"
Sep 09 15:54:21.014687 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:21.014651    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"]
Sep 09 15:54:21.015012 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:21.014978    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv" podUID="5778704d-c2e2-4553-b8a5-0a7ebf84f9dc" containerName="istio-proxy" containerID="cri-o://319dd4135318c95a948b2a0cf44a7df19c8a2f43dfe79a3b585be5c694798fc3" gracePeriod=30
Sep 09 15:54:26.290843 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.290768    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"
Sep 09 15:54:26.408447 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.408409    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-credential-socket\") pod \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\" (UID: \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\") "
Sep 09 15:54:26.408447 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.408449    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-istiod-ca-cert\") pod \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\" (UID: \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\") "
Sep 09 15:54:26.408702 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.408485    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-istio-envoy\") pod \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\" (UID: \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\") "
Sep 09 15:54:26.408702 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.408527    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-istio-podinfo\") pod \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\" (UID: \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\") "
Sep 09 15:54:26.408702 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.408556    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-istio-data\") pod \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\" (UID: \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\") "
Sep 09 15:54:26.408702 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.408626    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6xc5\" (UniqueName: \"kubernetes.io/projected/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-kube-api-access-s6xc5\") pod \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\" (UID: \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\") "
Sep 09 15:54:26.408702 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.408671    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-workload-certs\") pod \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\" (UID: \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\") "
Sep 09 15:54:26.408702 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.408694    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-istio-token\") pod \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\" (UID: \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\") "
Sep 09 15:54:26.409041 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.408761    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-workload-socket\") pod \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\" (UID: \"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc\") "
Sep 09 15:54:26.409041 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.408756    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5778704d-c2e2-4553-b8a5-0a7ebf84f9dc" (UID: "5778704d-c2e2-4553-b8a5-0a7ebf84f9dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:54:26.409041 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.408886    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5778704d-c2e2-4553-b8a5-0a7ebf84f9dc" (UID: "5778704d-c2e2-4553-b8a5-0a7ebf84f9dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:54:26.409204 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.409039    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5778704d-c2e2-4553-b8a5-0a7ebf84f9dc" (UID: "5778704d-c2e2-4553-b8a5-0a7ebf84f9dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:54:26.409204 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.409053    2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-credential-socket\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:26.409204 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.409071    2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-istiod-ca-cert\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:26.409204 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.409144    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-istio-data" (OuterVolumeSpecName: "istio-data") pod "5778704d-c2e2-4553-b8a5-0a7ebf84f9dc" (UID: "5778704d-c2e2-4553-b8a5-0a7ebf84f9dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:54:26.409412 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.409213    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5778704d-c2e2-4553-b8a5-0a7ebf84f9dc" (UID: "5778704d-c2e2-4553-b8a5-0a7ebf84f9dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:54:26.411282 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.411243    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-kube-api-access-s6xc5" (OuterVolumeSpecName: "kube-api-access-s6xc5") pod "5778704d-c2e2-4553-b8a5-0a7ebf84f9dc" (UID: "5778704d-c2e2-4553-b8a5-0a7ebf84f9dc"). InnerVolumeSpecName "kube-api-access-s6xc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:54:26.411409 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.411254    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5778704d-c2e2-4553-b8a5-0a7ebf84f9dc" (UID: "5778704d-c2e2-4553-b8a5-0a7ebf84f9dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:54:26.411409 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.411346    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-istio-token" (OuterVolumeSpecName: "istio-token") pod "5778704d-c2e2-4553-b8a5-0a7ebf84f9dc" (UID: "5778704d-c2e2-4553-b8a5-0a7ebf84f9dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:54:26.411659 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.411637    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5778704d-c2e2-4553-b8a5-0a7ebf84f9dc" (UID: "5778704d-c2e2-4553-b8a5-0a7ebf84f9dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 15:54:26.510394 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.510349    2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-istio-podinfo\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:26.510394 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.510380    2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-istio-data\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:26.510394 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.510394    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6xc5\" (UniqueName: \"kubernetes.io/projected/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-kube-api-access-s6xc5\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:26.510394 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.510404    2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-workload-certs\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:26.510767 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.510413    2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-istio-token\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:26.510767 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.510421    2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-workload-socket\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:26.510767 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.510430    2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc-istio-envoy\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:54:26.673420 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.673385    2553 generic.go:356] "Generic (PLEG): container finished" podID="5778704d-c2e2-4553-b8a5-0a7ebf84f9dc" containerID="319dd4135318c95a948b2a0cf44a7df19c8a2f43dfe79a3b585be5c694798fc3" exitCode=0
Sep 09 15:54:26.673634 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.673439    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"
Sep 09 15:54:26.673634 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.673468    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv" event={"ID":"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc","Type":"ContainerDied","Data":"319dd4135318c95a948b2a0cf44a7df19c8a2f43dfe79a3b585be5c694798fc3"}
Sep 09 15:54:26.673634 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.673556    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv" event={"ID":"5778704d-c2e2-4553-b8a5-0a7ebf84f9dc","Type":"ContainerDied","Data":"b0b5c688698a30843e1eda71e4a13f6cf7407bdcdefcaa3b56daee6951ba9f24"}
Sep 09 15:54:26.673634 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.673587    2553 scope.go:117] "RemoveContainer" containerID="319dd4135318c95a948b2a0cf44a7df19c8a2f43dfe79a3b585be5c694798fc3"
Sep 09 15:54:26.683563 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.683545    2553 scope.go:117] "RemoveContainer" containerID="b8833e8ec397320bddc2396cb9a39708c97e818883a51aea284024664971fa8b"
Sep 09 15:54:26.692223 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.692204    2553 scope.go:117] "RemoveContainer" containerID="319dd4135318c95a948b2a0cf44a7df19c8a2f43dfe79a3b585be5c694798fc3"
Sep 09 15:54:26.692539 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:54:26.692517    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"319dd4135318c95a948b2a0cf44a7df19c8a2f43dfe79a3b585be5c694798fc3\": container with ID starting with 319dd4135318c95a948b2a0cf44a7df19c8a2f43dfe79a3b585be5c694798fc3 not found: ID does not exist" containerID="319dd4135318c95a948b2a0cf44a7df19c8a2f43dfe79a3b585be5c694798fc3"
Sep 09 15:54:26.692613 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.692555    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"319dd4135318c95a948b2a0cf44a7df19c8a2f43dfe79a3b585be5c694798fc3"} err="failed to get container status \"319dd4135318c95a948b2a0cf44a7df19c8a2f43dfe79a3b585be5c694798fc3\": rpc error: code = NotFound desc = could not find container \"319dd4135318c95a948b2a0cf44a7df19c8a2f43dfe79a3b585be5c694798fc3\": container with ID starting with 319dd4135318c95a948b2a0cf44a7df19c8a2f43dfe79a3b585be5c694798fc3 not found: ID does not exist"
Sep 09 15:54:26.692613 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.692583    2553 scope.go:117] "RemoveContainer" containerID="b8833e8ec397320bddc2396cb9a39708c97e818883a51aea284024664971fa8b"
Sep 09 15:54:26.692893 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:54:26.692875    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8833e8ec397320bddc2396cb9a39708c97e818883a51aea284024664971fa8b\": container with ID starting with b8833e8ec397320bddc2396cb9a39708c97e818883a51aea284024664971fa8b not found: ID does not exist" containerID="b8833e8ec397320bddc2396cb9a39708c97e818883a51aea284024664971fa8b"
Sep 09 15:54:26.692958 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.692901    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8833e8ec397320bddc2396cb9a39708c97e818883a51aea284024664971fa8b"} err="failed to get container status \"b8833e8ec397320bddc2396cb9a39708c97e818883a51aea284024664971fa8b\": rpc error: code = NotFound desc = could not find container \"b8833e8ec397320bddc2396cb9a39708c97e818883a51aea284024664971fa8b\": container with ID starting with b8833e8ec397320bddc2396cb9a39708c97e818883a51aea284024664971fa8b not found: ID does not exist"
Sep 09 15:54:26.696052 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.696028    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"]
Sep 09 15:54:26.702027 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.702003    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-1e15c8-openshift-default-5d6989d8f8-kb2cv"]
Sep 09 15:54:26.963429 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:26.963353    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5778704d-c2e2-4553-b8a5-0a7ebf84f9dc" path="/var/lib/kubelet/pods/5778704d-c2e2-4553-b8a5-0a7ebf84f9dc/volumes"
Sep 09 15:54:35.291677 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:35.291644    2553 scope.go:117] "RemoveContainer" containerID="be089da702bd8dfdfa25390aa0cd05a00ae9306afd3916fb83794f997f0c1208"
Sep 09 15:54:35.301912 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:35.301888    2553 scope.go:117] "RemoveContainer" containerID="13f1abea79589731d7be57557aac915377b3ce68926e705d0583d2bbb049fe6b"
Sep 09 15:54:35.310033 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:35.310010    2553 scope.go:117] "RemoveContainer" containerID="53f4cec957b58b7e641fb97ed0a3ea04a86addd43dbdd026088bd4ccca26ea75"
Sep 09 15:54:50.453971 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.453929    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-482dce/sub-test-model-482dce-kserve-7d64cc68c-rfw7g"]
Sep 09 15:54:50.454421 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.454236    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5778704d-c2e2-4553-b8a5-0a7ebf84f9dc" containerName="istio-proxy"
Sep 09 15:54:50.454421 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.454248    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="5778704d-c2e2-4553-b8a5-0a7ebf84f9dc" containerName="istio-proxy"
Sep 09 15:54:50.454421 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.454259    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c70ce445-aedc-4429-8beb-0f21a23c357b" containerName="main"
Sep 09 15:54:50.454421 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.454265    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c70ce445-aedc-4429-8beb-0f21a23c357b" containerName="main"
Sep 09 15:54:50.454421 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.454282    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39cb9b9a-9e10-4879-ad05-99155842ca9e" containerName="payload-processing"
Sep 09 15:54:50.454421 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.454288    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="39cb9b9a-9e10-4879-ad05-99155842ca9e" containerName="payload-processing"
Sep 09 15:54:50.454421 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.454294    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c314dcd2-cca2-475e-8f5c-fdd62b477a98" containerName="revoke-keys"
Sep 09 15:54:50.454421 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.454298    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c314dcd2-cca2-475e-8f5c-fdd62b477a98" containerName="revoke-keys"
Sep 09 15:54:50.454421 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.454305    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd4ed05e-a093-4371-9e6b-6983225f91b5" containerName="main"
Sep 09 15:54:50.454421 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.454309    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd4ed05e-a093-4371-9e6b-6983225f91b5" containerName="main"
Sep 09 15:54:50.454421 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.454315    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1152d276-323d-4e1a-b67e-a2ce9927ae59" containerName="payload-pre-processing"
Sep 09 15:54:50.454421 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.454322    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="1152d276-323d-4e1a-b67e-a2ce9927ae59" containerName="payload-pre-processing"
Sep 09 15:54:50.454421 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.454332    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5778704d-c2e2-4553-b8a5-0a7ebf84f9dc" containerName="istio-proxy"
Sep 09 15:54:50.454421 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.454337    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="5778704d-c2e2-4553-b8a5-0a7ebf84f9dc" containerName="istio-proxy"
Sep 09 15:54:50.454421 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.454345    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc074101-6a71-45f8-8ba8-24a224e135bd" containerName="main"
Sep 09 15:54:50.454421 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.454349    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc074101-6a71-45f8-8ba8-24a224e135bd" containerName="main"
Sep 09 15:54:50.454421 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.454354    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7eeea333-89f5-4c49-bb04-6b860ef79ad8" containerName="main"
Sep 09 15:54:50.454421 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.454359    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="7eeea333-89f5-4c49-bb04-6b860ef79ad8" containerName="main"
Sep 09 15:54:50.454421 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.454367    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb2a60ce-3718-4c59-8c72-7608c97b6027" containerName="main"
Sep 09 15:54:50.454421 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.454372    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb2a60ce-3718-4c59-8c72-7608c97b6027" containerName="main"
Sep 09 15:54:50.454421 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.454422    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c314dcd2-cca2-475e-8f5c-fdd62b477a98" containerName="revoke-keys"
Sep 09 15:54:50.454421 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.454431    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="7eeea333-89f5-4c49-bb04-6b860ef79ad8" containerName="main"
Sep 09 15:54:50.455174 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.454437    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="1152d276-323d-4e1a-b67e-a2ce9927ae59" containerName="payload-pre-processing"
Sep 09 15:54:50.455174 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.454443    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="5778704d-c2e2-4553-b8a5-0a7ebf84f9dc" containerName="istio-proxy"
Sep 09 15:54:50.455174 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.454447    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="5778704d-c2e2-4553-b8a5-0a7ebf84f9dc" containerName="istio-proxy"
Sep 09 15:54:50.455174 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.454454    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="39cb9b9a-9e10-4879-ad05-99155842ca9e" containerName="payload-processing"
Sep 09 15:54:50.455174 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.454459    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd4ed05e-a093-4371-9e6b-6983225f91b5" containerName="main"
Sep 09 15:54:50.455174 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.454464    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb2a60ce-3718-4c59-8c72-7608c97b6027" containerName="main"
Sep 09 15:54:50.455174 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.454468    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc074101-6a71-45f8-8ba8-24a224e135bd" containerName="main"
Sep 09 15:54:50.455174 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.454473    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c70ce445-aedc-4429-8beb-0f21a23c357b" containerName="main"
Sep 09 15:54:50.457472 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.457450    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-482dce/sub-test-model-482dce-kserve-7d64cc68c-rfw7g"
Sep 09 15:54:50.460197 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.460174    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-482dce\"/\"openshift-service-ca.crt\""
Sep 09 15:54:50.461262 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.461241    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-482dce\"/\"kube-root-ca.crt\""
Sep 09 15:54:50.461402 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.461291    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-482dce\"/\"sub-test-model-482dce-kserve-self-signed-certs\""
Sep 09 15:54:50.461402 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.461372    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-482dce\"/\"default-dockercfg-95fv4\""
Sep 09 15:54:50.466233 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.466212    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-482dce/sub-test-model-482dce-kserve-7d64cc68c-rfw7g"]
Sep 09 15:54:50.535417 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.535383    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drwh8\" (UniqueName: \"kubernetes.io/projected/bc726883-7938-480a-a131-e46e7cfa36f3-kube-api-access-drwh8\") pod \"sub-test-model-482dce-kserve-7d64cc68c-rfw7g\" (UID: \"bc726883-7938-480a-a131-e46e7cfa36f3\") " pod="ai-tenant-e2e-shared-b-w3-482dce/sub-test-model-482dce-kserve-7d64cc68c-rfw7g"
Sep 09 15:54:50.535607 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.535430    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc726883-7938-480a-a131-e46e7cfa36f3-home\") pod \"sub-test-model-482dce-kserve-7d64cc68c-rfw7g\" (UID: \"bc726883-7938-480a-a131-e46e7cfa36f3\") " pod="ai-tenant-e2e-shared-b-w3-482dce/sub-test-model-482dce-kserve-7d64cc68c-rfw7g"
Sep 09 15:54:50.535607 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.535448    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc726883-7938-480a-a131-e46e7cfa36f3-tmp-dir\") pod \"sub-test-model-482dce-kserve-7d64cc68c-rfw7g\" (UID: \"bc726883-7938-480a-a131-e46e7cfa36f3\") " pod="ai-tenant-e2e-shared-b-w3-482dce/sub-test-model-482dce-kserve-7d64cc68c-rfw7g"
Sep 09 15:54:50.535607 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.535470    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc726883-7938-480a-a131-e46e7cfa36f3-dshm\") pod \"sub-test-model-482dce-kserve-7d64cc68c-rfw7g\" (UID: \"bc726883-7938-480a-a131-e46e7cfa36f3\") " pod="ai-tenant-e2e-shared-b-w3-482dce/sub-test-model-482dce-kserve-7d64cc68c-rfw7g"
Sep 09 15:54:50.535607 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.535558    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc726883-7938-480a-a131-e46e7cfa36f3-model-cache\") pod \"sub-test-model-482dce-kserve-7d64cc68c-rfw7g\" (UID: \"bc726883-7938-480a-a131-e46e7cfa36f3\") " pod="ai-tenant-e2e-shared-b-w3-482dce/sub-test-model-482dce-kserve-7d64cc68c-rfw7g"
Sep 09 15:54:50.535809 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.535633    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc726883-7938-480a-a131-e46e7cfa36f3-tls-certs\") pod \"sub-test-model-482dce-kserve-7d64cc68c-rfw7g\" (UID: \"bc726883-7938-480a-a131-e46e7cfa36f3\") " pod="ai-tenant-e2e-shared-b-w3-482dce/sub-test-model-482dce-kserve-7d64cc68c-rfw7g"
Sep 09 15:54:50.636329 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.636293    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc726883-7938-480a-a131-e46e7cfa36f3-dshm\") pod \"sub-test-model-482dce-kserve-7d64cc68c-rfw7g\" (UID: \"bc726883-7938-480a-a131-e46e7cfa36f3\") " pod="ai-tenant-e2e-shared-b-w3-482dce/sub-test-model-482dce-kserve-7d64cc68c-rfw7g"
Sep 09 15:54:50.636516 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.636336    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc726883-7938-480a-a131-e46e7cfa36f3-model-cache\") pod \"sub-test-model-482dce-kserve-7d64cc68c-rfw7g\" (UID: \"bc726883-7938-480a-a131-e46e7cfa36f3\") " pod="ai-tenant-e2e-shared-b-w3-482dce/sub-test-model-482dce-kserve-7d64cc68c-rfw7g"
Sep 09 15:54:50.636516 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.636383    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc726883-7938-480a-a131-e46e7cfa36f3-tls-certs\") pod \"sub-test-model-482dce-kserve-7d64cc68c-rfw7g\" (UID: \"bc726883-7938-480a-a131-e46e7cfa36f3\") " pod="ai-tenant-e2e-shared-b-w3-482dce/sub-test-model-482dce-kserve-7d64cc68c-rfw7g"
Sep 09 15:54:50.636516 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.636419    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drwh8\" (UniqueName: \"kubernetes.io/projected/bc726883-7938-480a-a131-e46e7cfa36f3-kube-api-access-drwh8\") pod \"sub-test-model-482dce-kserve-7d64cc68c-rfw7g\" (UID: \"bc726883-7938-480a-a131-e46e7cfa36f3\") " pod="ai-tenant-e2e-shared-b-w3-482dce/sub-test-model-482dce-kserve-7d64cc68c-rfw7g"
Sep 09 15:54:50.636516 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.636448    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc726883-7938-480a-a131-e46e7cfa36f3-home\") pod \"sub-test-model-482dce-kserve-7d64cc68c-rfw7g\" (UID: \"bc726883-7938-480a-a131-e46e7cfa36f3\") " pod="ai-tenant-e2e-shared-b-w3-482dce/sub-test-model-482dce-kserve-7d64cc68c-rfw7g"
Sep 09 15:54:50.636516 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.636473    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc726883-7938-480a-a131-e46e7cfa36f3-tmp-dir\") pod \"sub-test-model-482dce-kserve-7d64cc68c-rfw7g\" (UID: \"bc726883-7938-480a-a131-e46e7cfa36f3\") " pod="ai-tenant-e2e-shared-b-w3-482dce/sub-test-model-482dce-kserve-7d64cc68c-rfw7g"
Sep 09 15:54:50.636903 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.636870    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc726883-7938-480a-a131-e46e7cfa36f3-model-cache\") pod \"sub-test-model-482dce-kserve-7d64cc68c-rfw7g\" (UID: \"bc726883-7938-480a-a131-e46e7cfa36f3\") " pod="ai-tenant-e2e-shared-b-w3-482dce/sub-test-model-482dce-kserve-7d64cc68c-rfw7g"
Sep 09 15:54:50.637028 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.636909    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc726883-7938-480a-a131-e46e7cfa36f3-home\") pod \"sub-test-model-482dce-kserve-7d64cc68c-rfw7g\" (UID: \"bc726883-7938-480a-a131-e46e7cfa36f3\") " pod="ai-tenant-e2e-shared-b-w3-482dce/sub-test-model-482dce-kserve-7d64cc68c-rfw7g"
Sep 09 15:54:50.637028 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.636965    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc726883-7938-480a-a131-e46e7cfa36f3-tmp-dir\") pod \"sub-test-model-482dce-kserve-7d64cc68c-rfw7g\" (UID: \"bc726883-7938-480a-a131-e46e7cfa36f3\") " pod="ai-tenant-e2e-shared-b-w3-482dce/sub-test-model-482dce-kserve-7d64cc68c-rfw7g"
Sep 09 15:54:50.638616 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.638590    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc726883-7938-480a-a131-e46e7cfa36f3-dshm\") pod \"sub-test-model-482dce-kserve-7d64cc68c-rfw7g\" (UID: \"bc726883-7938-480a-a131-e46e7cfa36f3\") " pod="ai-tenant-e2e-shared-b-w3-482dce/sub-test-model-482dce-kserve-7d64cc68c-rfw7g"
Sep 09 15:54:50.639018 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.638999    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc726883-7938-480a-a131-e46e7cfa36f3-tls-certs\") pod \"sub-test-model-482dce-kserve-7d64cc68c-rfw7g\" (UID: \"bc726883-7938-480a-a131-e46e7cfa36f3\") " pod="ai-tenant-e2e-shared-b-w3-482dce/sub-test-model-482dce-kserve-7d64cc68c-rfw7g"
Sep 09 15:54:50.645124 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.645101    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drwh8\" (UniqueName: \"kubernetes.io/projected/bc726883-7938-480a-a131-e46e7cfa36f3-kube-api-access-drwh8\") pod \"sub-test-model-482dce-kserve-7d64cc68c-rfw7g\" (UID: \"bc726883-7938-480a-a131-e46e7cfa36f3\") " pod="ai-tenant-e2e-shared-b-w3-482dce/sub-test-model-482dce-kserve-7d64cc68c-rfw7g"
Sep 09 15:54:50.768336 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:50.768238    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-482dce/sub-test-model-482dce-kserve-7d64cc68c-rfw7g"
Sep 09 15:54:51.040287 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:51.040192    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-482dce/sub-test-model-482dce-kserve-7d64cc68c-rfw7g"]
Sep 09 15:54:51.043673 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:54:51.043647    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc726883_7938_480a_a131_e46e7cfa36f3.slice/crio-f9c817daba5dde3507aea089a39617535c923fa5e406d164e247f2213ec33197 WatchSource:0}: Error finding container f9c817daba5dde3507aea089a39617535c923fa5e406d164e247f2213ec33197: Status 404 returned error can't find the container with id f9c817daba5dde3507aea089a39617535c923fa5e406d164e247f2213ec33197
Sep 09 15:54:51.800694 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:51.800653    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-482dce/sub-test-model-482dce-kserve-7d64cc68c-rfw7g" event={"ID":"bc726883-7938-480a-a131-e46e7cfa36f3","Type":"ContainerStarted","Data":"dca09980dc2cd91f708258f943f9d842f5901e2cbe8495264d04a3a105e3ed64"}
Sep 09 15:54:51.801128 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:51.800701    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-482dce/sub-test-model-482dce-kserve-7d64cc68c-rfw7g" event={"ID":"bc726883-7938-480a-a131-e46e7cfa36f3","Type":"ContainerStarted","Data":"f9c817daba5dde3507aea089a39617535c923fa5e406d164e247f2213ec33197"}
Sep 09 15:54:51.819612 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:54:51.819559    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-482dce/sub-test-model-482dce-kserve-7d64cc68c-rfw7g" podStartSLOduration=1.819541782 podStartE2EDuration="1.819541782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:54:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:54:51.81736512 +0000 UTC m=+2057.416259766" watchObservedRunningTime="2026-09-09 15:54:51.819541782 +0000 UTC m=+2057.418436425"
Sep 09 15:55:00.769498 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:00.769398    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-482dce/sub-test-model-482dce-kserve-7d64cc68c-rfw7g"
Sep 09 15:55:00.769498 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:00.769467    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-482dce/sub-test-model-482dce-kserve-7d64cc68c-rfw7g"
Sep 09 15:55:00.782460 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:00.782430    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-482dce/sub-test-model-482dce-kserve-7d64cc68c-rfw7g"
Sep 09 15:55:00.857722 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:00.857687    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-482dce/sub-test-model-482dce-kserve-7d64cc68c-rfw7g"
Sep 09 15:55:32.641474 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:32.641430    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-482dce/sub-test-model-482dce-kserve-7d64cc68c-rfw7g"]
Sep 09 15:55:47.661560 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:47.661497    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-482dce/sub-test-model-482dce-kserve-7d64cc68c-rfw7g" podUID="bc726883-7938-480a-a131-e46e7cfa36f3" containerName="main" containerID="cri-o://dca09980dc2cd91f708258f943f9d842f5901e2cbe8495264d04a3a105e3ed64" gracePeriod=45
Sep 09 15:55:47.909772 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:47.909748    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-482dce/sub-test-model-482dce-kserve-7d64cc68c-rfw7g"
Sep 09 15:55:48.061925 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:48.061882    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drwh8\" (UniqueName: \"kubernetes.io/projected/bc726883-7938-480a-a131-e46e7cfa36f3-kube-api-access-drwh8\") pod \"bc726883-7938-480a-a131-e46e7cfa36f3\" (UID: \"bc726883-7938-480a-a131-e46e7cfa36f3\") "
Sep 09 15:55:48.062138 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:48.061946    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc726883-7938-480a-a131-e46e7cfa36f3-tmp-dir\") pod \"bc726883-7938-480a-a131-e46e7cfa36f3\" (UID: \"bc726883-7938-480a-a131-e46e7cfa36f3\") "
Sep 09 15:55:48.062138 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:48.061991    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc726883-7938-480a-a131-e46e7cfa36f3-model-cache\") pod \"bc726883-7938-480a-a131-e46e7cfa36f3\" (UID: \"bc726883-7938-480a-a131-e46e7cfa36f3\") "
Sep 09 15:55:48.062138 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:48.062025    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc726883-7938-480a-a131-e46e7cfa36f3-home\") pod \"bc726883-7938-480a-a131-e46e7cfa36f3\" (UID: \"bc726883-7938-480a-a131-e46e7cfa36f3\") "
Sep 09 15:55:48.062138 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:48.062049    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc726883-7938-480a-a131-e46e7cfa36f3-tls-certs\") pod \"bc726883-7938-480a-a131-e46e7cfa36f3\" (UID: \"bc726883-7938-480a-a131-e46e7cfa36f3\") "
Sep 09 15:55:48.062138 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:48.062082    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc726883-7938-480a-a131-e46e7cfa36f3-dshm\") pod \"bc726883-7938-480a-a131-e46e7cfa36f3\" (UID: \"bc726883-7938-480a-a131-e46e7cfa36f3\") "
Sep 09 15:55:48.062427 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:48.062253    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc726883-7938-480a-a131-e46e7cfa36f3-model-cache" (OuterVolumeSpecName: "model-cache") pod "bc726883-7938-480a-a131-e46e7cfa36f3" (UID: "bc726883-7938-480a-a131-e46e7cfa36f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:55:48.062427 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:48.062267    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc726883-7938-480a-a131-e46e7cfa36f3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bc726883-7938-480a-a131-e46e7cfa36f3" (UID: "bc726883-7938-480a-a131-e46e7cfa36f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:55:48.062581 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:48.062557    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc726883-7938-480a-a131-e46e7cfa36f3-home" (OuterVolumeSpecName: "home") pod "bc726883-7938-480a-a131-e46e7cfa36f3" (UID: "bc726883-7938-480a-a131-e46e7cfa36f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:55:48.064195 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:48.064160    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc726883-7938-480a-a131-e46e7cfa36f3-kube-api-access-drwh8" (OuterVolumeSpecName: "kube-api-access-drwh8") pod "bc726883-7938-480a-a131-e46e7cfa36f3" (UID: "bc726883-7938-480a-a131-e46e7cfa36f3"). InnerVolumeSpecName "kube-api-access-drwh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:55:48.064195 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:48.064179    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc726883-7938-480a-a131-e46e7cfa36f3-dshm" (OuterVolumeSpecName: "dshm") pod "bc726883-7938-480a-a131-e46e7cfa36f3" (UID: "bc726883-7938-480a-a131-e46e7cfa36f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:55:48.064338 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:48.064267    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc726883-7938-480a-a131-e46e7cfa36f3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bc726883-7938-480a-a131-e46e7cfa36f3" (UID: "bc726883-7938-480a-a131-e46e7cfa36f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:55:48.079114 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:48.079067    2553 generic.go:356] "Generic (PLEG): container finished" podID="bc726883-7938-480a-a131-e46e7cfa36f3" containerID="dca09980dc2cd91f708258f943f9d842f5901e2cbe8495264d04a3a105e3ed64" exitCode=0
Sep 09 15:55:48.079273 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:48.079225    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-482dce/sub-test-model-482dce-kserve-7d64cc68c-rfw7g" event={"ID":"bc726883-7938-480a-a131-e46e7cfa36f3","Type":"ContainerDied","Data":"dca09980dc2cd91f708258f943f9d842f5901e2cbe8495264d04a3a105e3ed64"}
Sep 09 15:55:48.079273 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:48.079264    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-482dce/sub-test-model-482dce-kserve-7d64cc68c-rfw7g" event={"ID":"bc726883-7938-480a-a131-e46e7cfa36f3","Type":"ContainerDied","Data":"f9c817daba5dde3507aea089a39617535c923fa5e406d164e247f2213ec33197"}
Sep 09 15:55:48.079385 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:48.079288    2553 scope.go:117] "RemoveContainer" containerID="dca09980dc2cd91f708258f943f9d842f5901e2cbe8495264d04a3a105e3ed64"
Sep 09 15:55:48.079506 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:48.079487    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-482dce/sub-test-model-482dce-kserve-7d64cc68c-rfw7g"
Sep 09 15:55:48.089217 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:48.089058    2553 scope.go:117] "RemoveContainer" containerID="dca09980dc2cd91f708258f943f9d842f5901e2cbe8495264d04a3a105e3ed64"
Sep 09 15:55:48.089462 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:55:48.089431    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dca09980dc2cd91f708258f943f9d842f5901e2cbe8495264d04a3a105e3ed64\": container with ID starting with dca09980dc2cd91f708258f943f9d842f5901e2cbe8495264d04a3a105e3ed64 not found: ID does not exist" containerID="dca09980dc2cd91f708258f943f9d842f5901e2cbe8495264d04a3a105e3ed64"
Sep 09 15:55:48.089540 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:48.089474    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dca09980dc2cd91f708258f943f9d842f5901e2cbe8495264d04a3a105e3ed64"} err="failed to get container status \"dca09980dc2cd91f708258f943f9d842f5901e2cbe8495264d04a3a105e3ed64\": rpc error: code = NotFound desc = could not find container \"dca09980dc2cd91f708258f943f9d842f5901e2cbe8495264d04a3a105e3ed64\": container with ID starting with dca09980dc2cd91f708258f943f9d842f5901e2cbe8495264d04a3a105e3ed64 not found: ID does not exist"
Sep 09 15:55:48.102479 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:48.102451    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-482dce/sub-test-model-482dce-kserve-7d64cc68c-rfw7g"]
Sep 09 15:55:48.108713 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:48.108682    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-482dce/sub-test-model-482dce-kserve-7d64cc68c-rfw7g"]
Sep 09 15:55:48.162937 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:48.162905    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc726883-7938-480a-a131-e46e7cfa36f3-home\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:55:48.163124 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:48.162954    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc726883-7938-480a-a131-e46e7cfa36f3-tls-certs\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:55:48.163124 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:48.162965    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc726883-7938-480a-a131-e46e7cfa36f3-dshm\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:55:48.163124 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:48.162975    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drwh8\" (UniqueName: \"kubernetes.io/projected/bc726883-7938-480a-a131-e46e7cfa36f3-kube-api-access-drwh8\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:55:48.163124 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:48.162985    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc726883-7938-480a-a131-e46e7cfa36f3-tmp-dir\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:55:48.163124 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:48.162997    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc726883-7938-480a-a131-e46e7cfa36f3-model-cache\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:55:48.962986 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:48.962953    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc726883-7938-480a-a131-e46e7cfa36f3" path="/var/lib/kubelet/pods/bc726883-7938-480a-a131-e46e7cfa36f3/volumes"
Sep 09 15:55:58.445023 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:58.444986    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-482dce/rate-test-model-482dce-kserve-6d9d5589f5-h6ckc"]
Sep 09 15:55:58.445385 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:58.445289    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc726883-7938-480a-a131-e46e7cfa36f3" containerName="main"
Sep 09 15:55:58.445385 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:58.445301    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc726883-7938-480a-a131-e46e7cfa36f3" containerName="main"
Sep 09 15:55:58.445385 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:58.445367    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc726883-7938-480a-a131-e46e7cfa36f3" containerName="main"
Sep 09 15:55:58.449828 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:58.449801    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-482dce/rate-test-model-482dce-kserve-6d9d5589f5-h6ckc"
Sep 09 15:55:58.454249 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:58.454210    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-482dce\"/\"kube-root-ca.crt\""
Sep 09 15:55:58.454249 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:58.454232    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-482dce\"/\"default-dockercfg-95fv4\""
Sep 09 15:55:58.454456 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:58.454257    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-482dce\"/\"openshift-service-ca.crt\""
Sep 09 15:55:58.454456 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:58.454270    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-482dce\"/\"rate-test-model-482dce-kserve-self-signed-certs\""
Sep 09 15:55:58.458879 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:58.458857    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-482dce/rate-test-model-482dce-kserve-6d9d5589f5-h6ckc"]
Sep 09 15:55:58.556262 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:58.556213    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6e31f56-405f-4693-a62e-e6bc8c3c3fd1-model-cache\") pod \"rate-test-model-482dce-kserve-6d9d5589f5-h6ckc\" (UID: \"c6e31f56-405f-4693-a62e-e6bc8c3c3fd1\") " pod="ai-tenant-e2e-shared-b-w3-482dce/rate-test-model-482dce-kserve-6d9d5589f5-h6ckc"
Sep 09 15:55:58.556462 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:58.556273    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvp87\" (UniqueName: \"kubernetes.io/projected/c6e31f56-405f-4693-a62e-e6bc8c3c3fd1-kube-api-access-cvp87\") pod \"rate-test-model-482dce-kserve-6d9d5589f5-h6ckc\" (UID: \"c6e31f56-405f-4693-a62e-e6bc8c3c3fd1\") " pod="ai-tenant-e2e-shared-b-w3-482dce/rate-test-model-482dce-kserve-6d9d5589f5-h6ckc"
Sep 09 15:55:58.556462 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:58.556308    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6e31f56-405f-4693-a62e-e6bc8c3c3fd1-home\") pod \"rate-test-model-482dce-kserve-6d9d5589f5-h6ckc\" (UID: \"c6e31f56-405f-4693-a62e-e6bc8c3c3fd1\") " pod="ai-tenant-e2e-shared-b-w3-482dce/rate-test-model-482dce-kserve-6d9d5589f5-h6ckc"
Sep 09 15:55:58.556462 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:58.556341    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6e31f56-405f-4693-a62e-e6bc8c3c3fd1-tmp-dir\") pod \"rate-test-model-482dce-kserve-6d9d5589f5-h6ckc\" (UID: \"c6e31f56-405f-4693-a62e-e6bc8c3c3fd1\") " pod="ai-tenant-e2e-shared-b-w3-482dce/rate-test-model-482dce-kserve-6d9d5589f5-h6ckc"
Sep 09 15:55:58.556462 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:58.556436    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6e31f56-405f-4693-a62e-e6bc8c3c3fd1-tls-certs\") pod \"rate-test-model-482dce-kserve-6d9d5589f5-h6ckc\" (UID: \"c6e31f56-405f-4693-a62e-e6bc8c3c3fd1\") " pod="ai-tenant-e2e-shared-b-w3-482dce/rate-test-model-482dce-kserve-6d9d5589f5-h6ckc"
Sep 09 15:55:58.556626 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:58.556546    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6e31f56-405f-4693-a62e-e6bc8c3c3fd1-dshm\") pod \"rate-test-model-482dce-kserve-6d9d5589f5-h6ckc\" (UID: \"c6e31f56-405f-4693-a62e-e6bc8c3c3fd1\") " pod="ai-tenant-e2e-shared-b-w3-482dce/rate-test-model-482dce-kserve-6d9d5589f5-h6ckc"
Sep 09 15:55:58.658054 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:58.658003    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6e31f56-405f-4693-a62e-e6bc8c3c3fd1-dshm\") pod \"rate-test-model-482dce-kserve-6d9d5589f5-h6ckc\" (UID: \"c6e31f56-405f-4693-a62e-e6bc8c3c3fd1\") " pod="ai-tenant-e2e-shared-b-w3-482dce/rate-test-model-482dce-kserve-6d9d5589f5-h6ckc"
Sep 09 15:55:58.658244 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:58.658075    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6e31f56-405f-4693-a62e-e6bc8c3c3fd1-model-cache\") pod \"rate-test-model-482dce-kserve-6d9d5589f5-h6ckc\" (UID: \"c6e31f56-405f-4693-a62e-e6bc8c3c3fd1\") " pod="ai-tenant-e2e-shared-b-w3-482dce/rate-test-model-482dce-kserve-6d9d5589f5-h6ckc"
Sep 09 15:55:58.658244 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:58.658097    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvp87\" (UniqueName: \"kubernetes.io/projected/c6e31f56-405f-4693-a62e-e6bc8c3c3fd1-kube-api-access-cvp87\") pod \"rate-test-model-482dce-kserve-6d9d5589f5-h6ckc\" (UID: \"c6e31f56-405f-4693-a62e-e6bc8c3c3fd1\") " pod="ai-tenant-e2e-shared-b-w3-482dce/rate-test-model-482dce-kserve-6d9d5589f5-h6ckc"
Sep 09 15:55:58.658244 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:58.658116    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6e31f56-405f-4693-a62e-e6bc8c3c3fd1-home\") pod \"rate-test-model-482dce-kserve-6d9d5589f5-h6ckc\" (UID: \"c6e31f56-405f-4693-a62e-e6bc8c3c3fd1\") " pod="ai-tenant-e2e-shared-b-w3-482dce/rate-test-model-482dce-kserve-6d9d5589f5-h6ckc"
Sep 09 15:55:58.658244 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:58.658137    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6e31f56-405f-4693-a62e-e6bc8c3c3fd1-tmp-dir\") pod \"rate-test-model-482dce-kserve-6d9d5589f5-h6ckc\" (UID: \"c6e31f56-405f-4693-a62e-e6bc8c3c3fd1\") " pod="ai-tenant-e2e-shared-b-w3-482dce/rate-test-model-482dce-kserve-6d9d5589f5-h6ckc"
Sep 09 15:55:58.658244 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:58.658155    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6e31f56-405f-4693-a62e-e6bc8c3c3fd1-tls-certs\") pod \"rate-test-model-482dce-kserve-6d9d5589f5-h6ckc\" (UID: \"c6e31f56-405f-4693-a62e-e6bc8c3c3fd1\") " pod="ai-tenant-e2e-shared-b-w3-482dce/rate-test-model-482dce-kserve-6d9d5589f5-h6ckc"
Sep 09 15:55:58.658593 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:58.658559    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6e31f56-405f-4693-a62e-e6bc8c3c3fd1-model-cache\") pod \"rate-test-model-482dce-kserve-6d9d5589f5-h6ckc\" (UID: \"c6e31f56-405f-4693-a62e-e6bc8c3c3fd1\") " pod="ai-tenant-e2e-shared-b-w3-482dce/rate-test-model-482dce-kserve-6d9d5589f5-h6ckc"
Sep 09 15:55:58.658593 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:58.658580    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6e31f56-405f-4693-a62e-e6bc8c3c3fd1-tmp-dir\") pod \"rate-test-model-482dce-kserve-6d9d5589f5-h6ckc\" (UID: \"c6e31f56-405f-4693-a62e-e6bc8c3c3fd1\") " pod="ai-tenant-e2e-shared-b-w3-482dce/rate-test-model-482dce-kserve-6d9d5589f5-h6ckc"
Sep 09 15:55:58.658794 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:58.658567    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6e31f56-405f-4693-a62e-e6bc8c3c3fd1-home\") pod \"rate-test-model-482dce-kserve-6d9d5589f5-h6ckc\" (UID: \"c6e31f56-405f-4693-a62e-e6bc8c3c3fd1\") " pod="ai-tenant-e2e-shared-b-w3-482dce/rate-test-model-482dce-kserve-6d9d5589f5-h6ckc"
Sep 09 15:55:58.660363 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:58.660342    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6e31f56-405f-4693-a62e-e6bc8c3c3fd1-dshm\") pod \"rate-test-model-482dce-kserve-6d9d5589f5-h6ckc\" (UID: \"c6e31f56-405f-4693-a62e-e6bc8c3c3fd1\") " pod="ai-tenant-e2e-shared-b-w3-482dce/rate-test-model-482dce-kserve-6d9d5589f5-h6ckc"
Sep 09 15:55:58.660616 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:58.660596    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6e31f56-405f-4693-a62e-e6bc8c3c3fd1-tls-certs\") pod \"rate-test-model-482dce-kserve-6d9d5589f5-h6ckc\" (UID: \"c6e31f56-405f-4693-a62e-e6bc8c3c3fd1\") " pod="ai-tenant-e2e-shared-b-w3-482dce/rate-test-model-482dce-kserve-6d9d5589f5-h6ckc"
Sep 09 15:55:58.666938 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:58.666912    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvp87\" (UniqueName: \"kubernetes.io/projected/c6e31f56-405f-4693-a62e-e6bc8c3c3fd1-kube-api-access-cvp87\") pod \"rate-test-model-482dce-kserve-6d9d5589f5-h6ckc\" (UID: \"c6e31f56-405f-4693-a62e-e6bc8c3c3fd1\") " pod="ai-tenant-e2e-shared-b-w3-482dce/rate-test-model-482dce-kserve-6d9d5589f5-h6ckc"
Sep 09 15:55:58.759692 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:58.759598    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-482dce/rate-test-model-482dce-kserve-6d9d5589f5-h6ckc"
Sep 09 15:55:59.031944 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:59.031919    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-482dce/rate-test-model-482dce-kserve-6d9d5589f5-h6ckc"]
Sep 09 15:55:59.034550 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:55:59.034523    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6e31f56_405f_4693_a62e_e6bc8c3c3fd1.slice/crio-546c39c5210fae3096de8be325f7c68847b55b6cb2e8d076dd6a11838c79b868 WatchSource:0}: Error finding container 546c39c5210fae3096de8be325f7c68847b55b6cb2e8d076dd6a11838c79b868: Status 404 returned error can't find the container with id 546c39c5210fae3096de8be325f7c68847b55b6cb2e8d076dd6a11838c79b868
Sep 09 15:55:59.135830 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:59.135783    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-482dce/rate-test-model-482dce-kserve-6d9d5589f5-h6ckc" event={"ID":"c6e31f56-405f-4693-a62e-e6bc8c3c3fd1","Type":"ContainerStarted","Data":"2c21857ed6b072bd4ab383dcfb415d5554b8c8e44a2e7b960d6906599911a9f5"}
Sep 09 15:55:59.136007 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:59.135838    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-482dce/rate-test-model-482dce-kserve-6d9d5589f5-h6ckc" event={"ID":"c6e31f56-405f-4693-a62e-e6bc8c3c3fd1","Type":"ContainerStarted","Data":"546c39c5210fae3096de8be325f7c68847b55b6cb2e8d076dd6a11838c79b868"}
Sep 09 15:55:59.153702 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:55:59.153643    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-482dce/rate-test-model-482dce-kserve-6d9d5589f5-h6ckc" podStartSLOduration=1.153622723 podStartE2EDuration="1.153622723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:55:59.152177242 +0000 UTC m=+2124.751071887" watchObservedRunningTime="2026-09-09 15:55:59.153622723 +0000 UTC m=+2124.752517371"
Sep 09 15:56:08.760725 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:08.760674    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-482dce/rate-test-model-482dce-kserve-6d9d5589f5-h6ckc"
Sep 09 15:56:08.760725 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:08.760755    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-482dce/rate-test-model-482dce-kserve-6d9d5589f5-h6ckc"
Sep 09 15:56:08.773670 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:08.773634    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-482dce/rate-test-model-482dce-kserve-6d9d5589f5-h6ckc"
Sep 09 15:56:09.197381 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:09.197353    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-482dce/rate-test-model-482dce-kserve-6d9d5589f5-h6ckc"
Sep 09 15:56:32.747171 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:32.747088    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-482dce/rate-test-model-482dce-kserve-6d9d5589f5-h6ckc"]
Sep 09 15:56:33.572115 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:33.572075    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"]
Sep 09 15:56:33.575466 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:33.575441    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"
Sep 09 15:56:33.578593 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:33.578570    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-1218fe-openshift-default-dockercfg-68dfw\""
Sep 09 15:56:33.586424 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:33.586389    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"]
Sep 09 15:56:33.681022 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:33.680974    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32496718-d42f-49cd-b024-9a98c0811354-istiod-ca-cert\") pod \"e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm\" (UID: \"32496718-d42f-49cd-b024-9a98c0811354\") " pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"
Sep 09 15:56:33.681224 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:33.681030    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32496718-d42f-49cd-b024-9a98c0811354-istio-envoy\") pod \"e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm\" (UID: \"32496718-d42f-49cd-b024-9a98c0811354\") " pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"
Sep 09 15:56:33.681224 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:33.681119    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9xd8\" (UniqueName: \"kubernetes.io/projected/32496718-d42f-49cd-b024-9a98c0811354-kube-api-access-t9xd8\") pod \"e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm\" (UID: \"32496718-d42f-49cd-b024-9a98c0811354\") " pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"
Sep 09 15:56:33.681224 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:33.681160    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32496718-d42f-49cd-b024-9a98c0811354-istio-token\") pod \"e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm\" (UID: \"32496718-d42f-49cd-b024-9a98c0811354\") " pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"
Sep 09 15:56:33.681224 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:33.681197    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32496718-d42f-49cd-b024-9a98c0811354-workload-socket\") pod \"e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm\" (UID: \"32496718-d42f-49cd-b024-9a98c0811354\") " pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"
Sep 09 15:56:33.681442 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:33.681226    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32496718-d42f-49cd-b024-9a98c0811354-istio-podinfo\") pod \"e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm\" (UID: \"32496718-d42f-49cd-b024-9a98c0811354\") " pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"
Sep 09 15:56:33.681442 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:33.681264    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32496718-d42f-49cd-b024-9a98c0811354-istio-data\") pod \"e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm\" (UID: \"32496718-d42f-49cd-b024-9a98c0811354\") " pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"
Sep 09 15:56:33.681442 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:33.681324    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32496718-d42f-49cd-b024-9a98c0811354-workload-certs\") pod \"e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm\" (UID: \"32496718-d42f-49cd-b024-9a98c0811354\") " pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"
Sep 09 15:56:33.681442 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:33.681394    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32496718-d42f-49cd-b024-9a98c0811354-credential-socket\") pod \"e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm\" (UID: \"32496718-d42f-49cd-b024-9a98c0811354\") " pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"
Sep 09 15:56:33.782185 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:33.782142    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32496718-d42f-49cd-b024-9a98c0811354-istiod-ca-cert\") pod \"e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm\" (UID: \"32496718-d42f-49cd-b024-9a98c0811354\") " pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"
Sep 09 15:56:33.782185 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:33.782185    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32496718-d42f-49cd-b024-9a98c0811354-istio-envoy\") pod \"e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm\" (UID: \"32496718-d42f-49cd-b024-9a98c0811354\") " pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"
Sep 09 15:56:33.782828 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:33.782214    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9xd8\" (UniqueName: \"kubernetes.io/projected/32496718-d42f-49cd-b024-9a98c0811354-kube-api-access-t9xd8\") pod \"e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm\" (UID: \"32496718-d42f-49cd-b024-9a98c0811354\") " pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"
Sep 09 15:56:33.782828 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:33.782239    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32496718-d42f-49cd-b024-9a98c0811354-istio-token\") pod \"e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm\" (UID: \"32496718-d42f-49cd-b024-9a98c0811354\") " pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"
Sep 09 15:56:33.782828 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:33.782266    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32496718-d42f-49cd-b024-9a98c0811354-workload-socket\") pod \"e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm\" (UID: \"32496718-d42f-49cd-b024-9a98c0811354\") " pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"
Sep 09 15:56:33.782828 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:33.782286    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32496718-d42f-49cd-b024-9a98c0811354-istio-podinfo\") pod \"e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm\" (UID: \"32496718-d42f-49cd-b024-9a98c0811354\") " pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"
Sep 09 15:56:33.782828 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:33.782321    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32496718-d42f-49cd-b024-9a98c0811354-istio-data\") pod \"e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm\" (UID: \"32496718-d42f-49cd-b024-9a98c0811354\") " pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"
Sep 09 15:56:33.782828 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:33.782370    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32496718-d42f-49cd-b024-9a98c0811354-workload-certs\") pod \"e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm\" (UID: \"32496718-d42f-49cd-b024-9a98c0811354\") " pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"
Sep 09 15:56:33.782828 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:33.782420    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32496718-d42f-49cd-b024-9a98c0811354-credential-socket\") pod \"e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm\" (UID: \"32496718-d42f-49cd-b024-9a98c0811354\") " pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"
Sep 09 15:56:33.783218 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:33.782833    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32496718-d42f-49cd-b024-9a98c0811354-istio-data\") pod \"e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm\" (UID: \"32496718-d42f-49cd-b024-9a98c0811354\") " pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"
Sep 09 15:56:33.783218 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:33.782895    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32496718-d42f-49cd-b024-9a98c0811354-credential-socket\") pod \"e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm\" (UID: \"32496718-d42f-49cd-b024-9a98c0811354\") " pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"
Sep 09 15:56:33.783218 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:33.782986    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32496718-d42f-49cd-b024-9a98c0811354-workload-certs\") pod \"e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm\" (UID: \"32496718-d42f-49cd-b024-9a98c0811354\") " pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"
Sep 09 15:56:33.783218 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:33.782992    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32496718-d42f-49cd-b024-9a98c0811354-workload-socket\") pod \"e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm\" (UID: \"32496718-d42f-49cd-b024-9a98c0811354\") " pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"
Sep 09 15:56:33.783218 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:33.783045    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32496718-d42f-49cd-b024-9a98c0811354-istiod-ca-cert\") pod \"e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm\" (UID: \"32496718-d42f-49cd-b024-9a98c0811354\") " pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"
Sep 09 15:56:33.784697 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:33.784676    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32496718-d42f-49cd-b024-9a98c0811354-istio-envoy\") pod \"e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm\" (UID: \"32496718-d42f-49cd-b024-9a98c0811354\") " pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"
Sep 09 15:56:33.785117 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:33.785094    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32496718-d42f-49cd-b024-9a98c0811354-istio-podinfo\") pod \"e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm\" (UID: \"32496718-d42f-49cd-b024-9a98c0811354\") " pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"
Sep 09 15:56:33.790874 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:33.790845    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32496718-d42f-49cd-b024-9a98c0811354-istio-token\") pod \"e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm\" (UID: \"32496718-d42f-49cd-b024-9a98c0811354\") " pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"
Sep 09 15:56:33.790984 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:33.790914    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9xd8\" (UniqueName: \"kubernetes.io/projected/32496718-d42f-49cd-b024-9a98c0811354-kube-api-access-t9xd8\") pod \"e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm\" (UID: \"32496718-d42f-49cd-b024-9a98c0811354\") " pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"
Sep 09 15:56:33.887556 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:33.887471    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"
Sep 09 15:56:34.166502 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:34.166371    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"]
Sep 09 15:56:34.169488 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:56:34.169460    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32496718_d42f_49cd_b024_9a98c0811354.slice/crio-499890a8595a741c901c1a8e9d7c093cc4cdb30da82d8b23a831ffc178a096ee WatchSource:0}: Error finding container 499890a8595a741c901c1a8e9d7c093cc4cdb30da82d8b23a831ffc178a096ee: Status 404 returned error can't find the container with id 499890a8595a741c901c1a8e9d7c093cc4cdb30da82d8b23a831ffc178a096ee
Sep 09 15:56:34.172056 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:34.172022    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:56:34.172147 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:34.172093    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:56:34.172188 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:34.172151    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:56:34.312833 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:34.312799    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm" event={"ID":"32496718-d42f-49cd-b024-9a98c0811354","Type":"ContainerStarted","Data":"00dad01c3282c9bef05cd83c4e6ff033383083a9befeba9819394c0d55df9535"}
Sep 09 15:56:34.313025 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:34.312860    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm" event={"ID":"32496718-d42f-49cd-b024-9a98c0811354","Type":"ContainerStarted","Data":"499890a8595a741c901c1a8e9d7c093cc4cdb30da82d8b23a831ffc178a096ee"}
Sep 09 15:56:34.332515 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:34.332425    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm" podStartSLOduration=1.332403364 podStartE2EDuration="1.332403364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:56:34.33175689 +0000 UTC m=+2159.930651533" watchObservedRunningTime="2026-09-09 15:56:34.332403364 +0000 UTC m=+2159.931298010"
Sep 09 15:56:34.888619 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:34.888576    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"
Sep 09 15:56:35.365132 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:35.365093    2553 scope.go:117] "RemoveContainer" containerID="9c943d1f1b5756ef18e58f2f738a598f50f926cb8f761945aaa1a242a39c5c2c"
Sep 09 15:56:35.892818 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:35.892782    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"
Sep 09 15:56:36.323504 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:36.323470    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"
Sep 09 15:56:36.324437 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:36.324418    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"
Sep 09 15:56:38.322033 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:38.322000    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1218fe-cc75f9b58-v28rq"]
Sep 09 15:56:38.326658 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:38.326637    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1218fe-cc75f9b58-v28rq"
Sep 09 15:56:38.329452 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:38.329426    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-1218fe-dockercfg-lhcqj\""
Sep 09 15:56:38.329611 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:38.329453    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-1218fe\""
Sep 09 15:56:38.336482 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:38.336457    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1218fe-cc75f9b58-v28rq"]
Sep 09 15:56:38.356760 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:38.356714    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1218fe-8685d765b7-ph7rh"]
Sep 09 15:56:38.359982 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:38.359957    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1218fe-8685d765b7-ph7rh"
Sep 09 15:56:38.369305 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:38.369278    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1218fe-8685d765b7-ph7rh"]
Sep 09 15:56:38.429161 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:38.429122    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3bfd8e3-2dc5-45e5-83c8-af854c0354c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1218fe-cc75f9b58-v28rq\" (UID: \"d3bfd8e3-2dc5-45e5-83c8-af854c0354c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1218fe-cc75f9b58-v28rq"
Sep 09 15:56:38.429359 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:38.429220    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfhq4\" (UniqueName: \"kubernetes.io/projected/d3bfd8e3-2dc5-45e5-83c8-af854c0354c5-kube-api-access-lfhq4\") pod \"payload-pre-processing-e2e-ipp-a-1218fe-cc75f9b58-v28rq\" (UID: \"d3bfd8e3-2dc5-45e5-83c8-af854c0354c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1218fe-cc75f9b58-v28rq"
Sep 09 15:56:38.530857 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:38.530820    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f9365a6-cea0-4111-b12c-df9e0abd260e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1218fe-8685d765b7-ph7rh\" (UID: \"7f9365a6-cea0-4111-b12c-df9e0abd260e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1218fe-8685d765b7-ph7rh"
Sep 09 15:56:38.530857 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:38.530861    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3bfd8e3-2dc5-45e5-83c8-af854c0354c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1218fe-cc75f9b58-v28rq\" (UID: \"d3bfd8e3-2dc5-45e5-83c8-af854c0354c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1218fe-cc75f9b58-v28rq"
Sep 09 15:56:38.531118 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:38.530881    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkct4\" (UniqueName: \"kubernetes.io/projected/7f9365a6-cea0-4111-b12c-df9e0abd260e-kube-api-access-zkct4\") pod \"payload-processing-e2e-ipp-a-1218fe-8685d765b7-ph7rh\" (UID: \"7f9365a6-cea0-4111-b12c-df9e0abd260e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1218fe-8685d765b7-ph7rh"
Sep 09 15:56:38.531118 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:38.531027    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfhq4\" (UniqueName: \"kubernetes.io/projected/d3bfd8e3-2dc5-45e5-83c8-af854c0354c5-kube-api-access-lfhq4\") pod \"payload-pre-processing-e2e-ipp-a-1218fe-cc75f9b58-v28rq\" (UID: \"d3bfd8e3-2dc5-45e5-83c8-af854c0354c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1218fe-cc75f9b58-v28rq"
Sep 09 15:56:38.531652 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:38.531628    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3bfd8e3-2dc5-45e5-83c8-af854c0354c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1218fe-cc75f9b58-v28rq\" (UID: \"d3bfd8e3-2dc5-45e5-83c8-af854c0354c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1218fe-cc75f9b58-v28rq"
Sep 09 15:56:38.539403 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:38.539371    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfhq4\" (UniqueName: \"kubernetes.io/projected/d3bfd8e3-2dc5-45e5-83c8-af854c0354c5-kube-api-access-lfhq4\") pod \"payload-pre-processing-e2e-ipp-a-1218fe-cc75f9b58-v28rq\" (UID: \"d3bfd8e3-2dc5-45e5-83c8-af854c0354c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1218fe-cc75f9b58-v28rq"
Sep 09 15:56:38.632314 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:38.632219    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f9365a6-cea0-4111-b12c-df9e0abd260e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1218fe-8685d765b7-ph7rh\" (UID: \"7f9365a6-cea0-4111-b12c-df9e0abd260e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1218fe-8685d765b7-ph7rh"
Sep 09 15:56:38.632314 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:38.632263    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkct4\" (UniqueName: \"kubernetes.io/projected/7f9365a6-cea0-4111-b12c-df9e0abd260e-kube-api-access-zkct4\") pod \"payload-processing-e2e-ipp-a-1218fe-8685d765b7-ph7rh\" (UID: \"7f9365a6-cea0-4111-b12c-df9e0abd260e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1218fe-8685d765b7-ph7rh"
Sep 09 15:56:38.633030 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:38.633007    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f9365a6-cea0-4111-b12c-df9e0abd260e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1218fe-8685d765b7-ph7rh\" (UID: \"7f9365a6-cea0-4111-b12c-df9e0abd260e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1218fe-8685d765b7-ph7rh"
Sep 09 15:56:38.639906 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:38.639871    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1218fe-cc75f9b58-v28rq"
Sep 09 15:56:38.640904 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:38.640879    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkct4\" (UniqueName: \"kubernetes.io/projected/7f9365a6-cea0-4111-b12c-df9e0abd260e-kube-api-access-zkct4\") pod \"payload-processing-e2e-ipp-a-1218fe-8685d765b7-ph7rh\" (UID: \"7f9365a6-cea0-4111-b12c-df9e0abd260e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1218fe-8685d765b7-ph7rh"
Sep 09 15:56:38.671481 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:38.671445    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1218fe-8685d765b7-ph7rh"
Sep 09 15:56:38.912003 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:38.911974    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1218fe-cc75f9b58-v28rq"]
Sep 09 15:56:38.913362 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:56:38.913332    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3bfd8e3_2dc5_45e5_83c8_af854c0354c5.slice/crio-df6198cf538d8d47cddfe70d0d945f3f044ebf4d6a485a3e62d490e2d93774b5 WatchSource:0}: Error finding container df6198cf538d8d47cddfe70d0d945f3f044ebf4d6a485a3e62d490e2d93774b5: Status 404 returned error can't find the container with id df6198cf538d8d47cddfe70d0d945f3f044ebf4d6a485a3e62d490e2d93774b5
Sep 09 15:56:38.957757 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:56:38.957701    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f9365a6_cea0_4111_b12c_df9e0abd260e.slice/crio-f7fe9b3f30ec273f7d3f17ba3e2cb8da39c66d8834393173905889f1a7d2687d WatchSource:0}: Error finding container f7fe9b3f30ec273f7d3f17ba3e2cb8da39c66d8834393173905889f1a7d2687d: Status 404 returned error can't find the container with id f7fe9b3f30ec273f7d3f17ba3e2cb8da39c66d8834393173905889f1a7d2687d
Sep 09 15:56:38.964847 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:38.964824    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1218fe-8685d765b7-ph7rh"]
Sep 09 15:56:39.341844 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:39.341802    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1218fe-8685d765b7-ph7rh" event={"ID":"7f9365a6-cea0-4111-b12c-df9e0abd260e","Type":"ContainerStarted","Data":"af5229a4d1cf66a38b80722051c200ebcbde5426c72ed3c5a5792e027cb3642c"}
Sep 09 15:56:39.342334 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:39.341877    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1218fe-8685d765b7-ph7rh" event={"ID":"7f9365a6-cea0-4111-b12c-df9e0abd260e","Type":"ContainerStarted","Data":"f7fe9b3f30ec273f7d3f17ba3e2cb8da39c66d8834393173905889f1a7d2687d"}
Sep 09 15:56:39.342334 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:39.342128    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1218fe-8685d765b7-ph7rh"
Sep 09 15:56:39.343840 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:39.343811    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1218fe-cc75f9b58-v28rq" event={"ID":"d3bfd8e3-2dc5-45e5-83c8-af854c0354c5","Type":"ContainerStarted","Data":"dbb21718336261f8d834513915645c1e8f363db6405eecbd93c0e4eae3889133"}
Sep 09 15:56:39.343982 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:39.343858    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1218fe-cc75f9b58-v28rq" event={"ID":"d3bfd8e3-2dc5-45e5-83c8-af854c0354c5","Type":"ContainerStarted","Data":"df6198cf538d8d47cddfe70d0d945f3f044ebf4d6a485a3e62d490e2d93774b5"}
Sep 09 15:56:39.343982 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:39.343886    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1218fe-cc75f9b58-v28rq"
Sep 09 15:56:39.370579 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:39.370527    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-1218fe-8685d765b7-ph7rh" podStartSLOduration=1.370510089 podStartE2EDuration="1.370510089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:56:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:56:39.368725504 +0000 UTC m=+2164.967620149" watchObservedRunningTime="2026-09-09 15:56:39.370510089 +0000 UTC m=+2164.969404782"
Sep 09 15:56:39.389703 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:39.389650    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1218fe-cc75f9b58-v28rq" podStartSLOduration=1.389633162 podStartE2EDuration="1.389633162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:56:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:56:39.388530399 +0000 UTC m=+2164.987425055" watchObservedRunningTime="2026-09-09 15:56:39.389633162 +0000 UTC m=+2164.988527805"
Sep 09 15:56:47.766917 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:47.766865    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-482dce/rate-test-model-482dce-kserve-6d9d5589f5-h6ckc" podUID="c6e31f56-405f-4693-a62e-e6bc8c3c3fd1" containerName="main" containerID="cri-o://2c21857ed6b072bd4ab383dcfb415d5554b8c8e44a2e7b960d6906599911a9f5" gracePeriod=45
Sep 09 15:56:48.036352 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:48.036325    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-482dce/rate-test-model-482dce-kserve-6d9d5589f5-h6ckc"
Sep 09 15:56:48.122779 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:48.122722    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6e31f56-405f-4693-a62e-e6bc8c3c3fd1-tmp-dir\") pod \"c6e31f56-405f-4693-a62e-e6bc8c3c3fd1\" (UID: \"c6e31f56-405f-4693-a62e-e6bc8c3c3fd1\") "
Sep 09 15:56:48.122990 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:48.122797    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6e31f56-405f-4693-a62e-e6bc8c3c3fd1-dshm\") pod \"c6e31f56-405f-4693-a62e-e6bc8c3c3fd1\" (UID: \"c6e31f56-405f-4693-a62e-e6bc8c3c3fd1\") "
Sep 09 15:56:48.122990 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:48.122829    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvp87\" (UniqueName: \"kubernetes.io/projected/c6e31f56-405f-4693-a62e-e6bc8c3c3fd1-kube-api-access-cvp87\") pod \"c6e31f56-405f-4693-a62e-e6bc8c3c3fd1\" (UID: \"c6e31f56-405f-4693-a62e-e6bc8c3c3fd1\") "
Sep 09 15:56:48.122990 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:48.122904    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6e31f56-405f-4693-a62e-e6bc8c3c3fd1-tls-certs\") pod \"c6e31f56-405f-4693-a62e-e6bc8c3c3fd1\" (UID: \"c6e31f56-405f-4693-a62e-e6bc8c3c3fd1\") "
Sep 09 15:56:48.122990 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:48.122945    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e31f56-405f-4693-a62e-e6bc8c3c3fd1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c6e31f56-405f-4693-a62e-e6bc8c3c3fd1" (UID: "c6e31f56-405f-4693-a62e-e6bc8c3c3fd1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:56:48.122990 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:48.122957    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6e31f56-405f-4693-a62e-e6bc8c3c3fd1-model-cache\") pod \"c6e31f56-405f-4693-a62e-e6bc8c3c3fd1\" (UID: \"c6e31f56-405f-4693-a62e-e6bc8c3c3fd1\") "
Sep 09 15:56:48.123268 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:48.123081    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6e31f56-405f-4693-a62e-e6bc8c3c3fd1-home\") pod \"c6e31f56-405f-4693-a62e-e6bc8c3c3fd1\" (UID: \"c6e31f56-405f-4693-a62e-e6bc8c3c3fd1\") "
Sep 09 15:56:48.123268 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:48.123205    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e31f56-405f-4693-a62e-e6bc8c3c3fd1-model-cache" (OuterVolumeSpecName: "model-cache") pod "c6e31f56-405f-4693-a62e-e6bc8c3c3fd1" (UID: "c6e31f56-405f-4693-a62e-e6bc8c3c3fd1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:56:48.123382 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:48.123285    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e31f56-405f-4693-a62e-e6bc8c3c3fd1-home" (OuterVolumeSpecName: "home") pod "c6e31f56-405f-4693-a62e-e6bc8c3c3fd1" (UID: "c6e31f56-405f-4693-a62e-e6bc8c3c3fd1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:56:48.123451 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:48.123397    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6e31f56-405f-4693-a62e-e6bc8c3c3fd1-model-cache\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:56:48.123451 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:48.123413    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6e31f56-405f-4693-a62e-e6bc8c3c3fd1-home\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:56:48.123451 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:48.123423    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6e31f56-405f-4693-a62e-e6bc8c3c3fd1-tmp-dir\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:56:48.125054 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:48.125024    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e31f56-405f-4693-a62e-e6bc8c3c3fd1-dshm" (OuterVolumeSpecName: "dshm") pod "c6e31f56-405f-4693-a62e-e6bc8c3c3fd1" (UID: "c6e31f56-405f-4693-a62e-e6bc8c3c3fd1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:56:48.125054 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:48.125024    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6e31f56-405f-4693-a62e-e6bc8c3c3fd1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c6e31f56-405f-4693-a62e-e6bc8c3c3fd1" (UID: "c6e31f56-405f-4693-a62e-e6bc8c3c3fd1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:56:48.125201 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:48.125070    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6e31f56-405f-4693-a62e-e6bc8c3c3fd1-kube-api-access-cvp87" (OuterVolumeSpecName: "kube-api-access-cvp87") pod "c6e31f56-405f-4693-a62e-e6bc8c3c3fd1" (UID: "c6e31f56-405f-4693-a62e-e6bc8c3c3fd1"). InnerVolumeSpecName "kube-api-access-cvp87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:56:48.224848 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:48.224791    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6e31f56-405f-4693-a62e-e6bc8c3c3fd1-dshm\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:56:48.224848 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:48.224842    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvp87\" (UniqueName: \"kubernetes.io/projected/c6e31f56-405f-4693-a62e-e6bc8c3c3fd1-kube-api-access-cvp87\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:56:48.224848 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:48.224853    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6e31f56-405f-4693-a62e-e6bc8c3c3fd1-tls-certs\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:56:48.392517 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:48.392418    2553 generic.go:356] "Generic (PLEG): container finished" podID="c6e31f56-405f-4693-a62e-e6bc8c3c3fd1" containerID="2c21857ed6b072bd4ab383dcfb415d5554b8c8e44a2e7b960d6906599911a9f5" exitCode=0
Sep 09 15:56:48.392517 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:48.392470    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-482dce/rate-test-model-482dce-kserve-6d9d5589f5-h6ckc"
Sep 09 15:56:48.392815 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:48.392508    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-482dce/rate-test-model-482dce-kserve-6d9d5589f5-h6ckc" event={"ID":"c6e31f56-405f-4693-a62e-e6bc8c3c3fd1","Type":"ContainerDied","Data":"2c21857ed6b072bd4ab383dcfb415d5554b8c8e44a2e7b960d6906599911a9f5"}
Sep 09 15:56:48.392815 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:48.392558    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-482dce/rate-test-model-482dce-kserve-6d9d5589f5-h6ckc" event={"ID":"c6e31f56-405f-4693-a62e-e6bc8c3c3fd1","Type":"ContainerDied","Data":"546c39c5210fae3096de8be325f7c68847b55b6cb2e8d076dd6a11838c79b868"}
Sep 09 15:56:48.392815 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:48.392579    2553 scope.go:117] "RemoveContainer" containerID="2c21857ed6b072bd4ab383dcfb415d5554b8c8e44a2e7b960d6906599911a9f5"
Sep 09 15:56:48.401638 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:48.401621    2553 scope.go:117] "RemoveContainer" containerID="2c21857ed6b072bd4ab383dcfb415d5554b8c8e44a2e7b960d6906599911a9f5"
Sep 09 15:56:48.401984 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:56:48.401950    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c21857ed6b072bd4ab383dcfb415d5554b8c8e44a2e7b960d6906599911a9f5\": container with ID starting with 2c21857ed6b072bd4ab383dcfb415d5554b8c8e44a2e7b960d6906599911a9f5 not found: ID does not exist" containerID="2c21857ed6b072bd4ab383dcfb415d5554b8c8e44a2e7b960d6906599911a9f5"
Sep 09 15:56:48.402039 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:48.401993    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c21857ed6b072bd4ab383dcfb415d5554b8c8e44a2e7b960d6906599911a9f5"} err="failed to get container status \"2c21857ed6b072bd4ab383dcfb415d5554b8c8e44a2e7b960d6906599911a9f5\": rpc error: code = NotFound desc = could not find container \"2c21857ed6b072bd4ab383dcfb415d5554b8c8e44a2e7b960d6906599911a9f5\": container with ID starting with 2c21857ed6b072bd4ab383dcfb415d5554b8c8e44a2e7b960d6906599911a9f5 not found: ID does not exist"
Sep 09 15:56:48.413463 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:48.413435    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-482dce/rate-test-model-482dce-kserve-6d9d5589f5-h6ckc"]
Sep 09 15:56:48.417772 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:48.417750    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-482dce/rate-test-model-482dce-kserve-6d9d5589f5-h6ckc"]
Sep 09 15:56:48.962421 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:48.962387    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6e31f56-405f-4693-a62e-e6bc8c3c3fd1" path="/var/lib/kubelet/pods/c6e31f56-405f-4693-a62e-e6bc8c3c3fd1/volumes"
Sep 09 15:56:50.351666 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:50.351633    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1218fe-8685d765b7-ph7rh"
Sep 09 15:56:50.352140 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:50.351756    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1218fe-cc75f9b58-v28rq"
Sep 09 15:56:51.734427 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:51.733887    2553 patch_prober.go:28] interesting pod/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": read tcp 10.134.0.2:33538->10.134.0.82:15021: read: connection reset by peer" start-of-body=
Sep 09 15:56:51.734427 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:51.733956    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm" podUID="32496718-d42f-49cd-b024-9a98c0811354" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": read tcp 10.134.0.2:33538->10.134.0.82:15021: read: connection reset by peer"
Sep 09 15:56:52.417848 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:52.417815    2553 generic.go:356] "Generic (PLEG): container finished" podID="32496718-d42f-49cd-b024-9a98c0811354" containerID="00dad01c3282c9bef05cd83c4e6ff033383083a9befeba9819394c0d55df9535" exitCode=0
Sep 09 15:56:52.418031 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:52.417889    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm" event={"ID":"32496718-d42f-49cd-b024-9a98c0811354","Type":"ContainerDied","Data":"00dad01c3282c9bef05cd83c4e6ff033383083a9befeba9819394c0d55df9535"}
Sep 09 15:56:52.418315 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:52.418301    2553 scope.go:117] "RemoveContainer" containerID="00dad01c3282c9bef05cd83c4e6ff033383083a9befeba9819394c0d55df9535"
Sep 09 15:56:52.419404 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:52.419370    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:56:52.419507 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:52.419445    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:56:52.419507 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:52.419481    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:56:53.424459 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:53.424426    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm" event={"ID":"32496718-d42f-49cd-b024-9a98c0811354","Type":"ContainerStarted","Data":"baa05a173beb9194e3f70071b0dd6d52a84478fbb9a2c9058c2110b6e8d20509"}
Sep 09 15:56:53.888192 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:53.888098    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"
Sep 09 15:56:53.888368 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:53.888343    2553 patch_prober.go:28] interesting pod/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Sep 09 15:56:53.888423 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:53.888397    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm" podUID="32496718-d42f-49cd-b024-9a98c0811354" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Sep 09 15:56:54.888136 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:54.888097    2553 patch_prober.go:28] interesting pod/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Sep 09 15:56:54.888547 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:54.888155    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm" podUID="32496718-d42f-49cd-b024-9a98c0811354" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Sep 09 15:56:55.888560 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:55.888522    2553 patch_prober.go:28] interesting pod/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Sep 09 15:56:55.889088 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:55.888592    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm" podUID="32496718-d42f-49cd-b024-9a98c0811354" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Sep 09 15:56:56.888509 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:56.888466    2553 patch_prober.go:28] interesting pod/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Sep 09 15:56:56.888704 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:56.888528    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm" podUID="32496718-d42f-49cd-b024-9a98c0811354" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Sep 09 15:56:57.888282 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:57.888235    2553 patch_prober.go:28] interesting pod/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Sep 09 15:56:57.888483 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:57.888305    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm" podUID="32496718-d42f-49cd-b024-9a98c0811354" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Sep 09 15:56:58.892960 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:58.892928    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"
Sep 09 15:56:58.893339 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:58.893193    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"
Sep 09 15:56:58.893816 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:56:58.893800    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"
Sep 09 15:57:07.761983 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:07.761949    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a2b442-7887cc56f8-xkk4q"]
Sep 09 15:57:07.762383 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:07.762293    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6e31f56-405f-4693-a62e-e6bc8c3c3fd1" containerName="main"
Sep 09 15:57:07.762383 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:07.762305    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6e31f56-405f-4693-a62e-e6bc8c3c3fd1" containerName="main"
Sep 09 15:57:07.762383 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:07.762378    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6e31f56-405f-4693-a62e-e6bc8c3c3fd1" containerName="main"
Sep 09 15:57:07.766467 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:07.766447    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2b442-7887cc56f8-xkk4q"
Sep 09 15:57:07.769598 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:07.769570    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a2b442-dockercfg-zn6qs\""
Sep 09 15:57:07.769598 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:07.769589    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a2b442\""
Sep 09 15:57:07.778667 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:07.778644    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a2b442-7887cc56f8-xkk4q"]
Sep 09 15:57:07.807322 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:07.807286    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a2b442-9cbbd6765-4qnxr"]
Sep 09 15:57:07.810662 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:07.810645    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2b442-9cbbd6765-4qnxr"
Sep 09 15:57:07.820401 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:07.820376    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a2b442-9cbbd6765-4qnxr"]
Sep 09 15:57:07.906279 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:07.906245    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xndnj\" (UniqueName: \"kubernetes.io/projected/d74849df-dcca-410f-93f2-7b84f72e6b59-kube-api-access-xndnj\") pod \"payload-pre-processing-e2e-ipp-b-a2b442-7887cc56f8-xkk4q\" (UID: \"d74849df-dcca-410f-93f2-7b84f72e6b59\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2b442-7887cc56f8-xkk4q"
Sep 09 15:57:07.906456 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:07.906290    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d74849df-dcca-410f-93f2-7b84f72e6b59-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a2b442-7887cc56f8-xkk4q\" (UID: \"d74849df-dcca-410f-93f2-7b84f72e6b59\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2b442-7887cc56f8-xkk4q"
Sep 09 15:57:07.906456 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:07.906342    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shqmj\" (UniqueName: \"kubernetes.io/projected/0452b747-ebc3-4807-87c3-927b2be91b08-kube-api-access-shqmj\") pod \"payload-processing-e2e-ipp-b-a2b442-9cbbd6765-4qnxr\" (UID: \"0452b747-ebc3-4807-87c3-927b2be91b08\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a2b442-9cbbd6765-4qnxr"
Sep 09 15:57:07.906456 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:07.906441    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0452b747-ebc3-4807-87c3-927b2be91b08-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a2b442-9cbbd6765-4qnxr\" (UID: \"0452b747-ebc3-4807-87c3-927b2be91b08\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a2b442-9cbbd6765-4qnxr"
Sep 09 15:57:08.007545 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:08.007510    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0452b747-ebc3-4807-87c3-927b2be91b08-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a2b442-9cbbd6765-4qnxr\" (UID: \"0452b747-ebc3-4807-87c3-927b2be91b08\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a2b442-9cbbd6765-4qnxr"
Sep 09 15:57:08.007718 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:08.007560    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xndnj\" (UniqueName: \"kubernetes.io/projected/d74849df-dcca-410f-93f2-7b84f72e6b59-kube-api-access-xndnj\") pod \"payload-pre-processing-e2e-ipp-b-a2b442-7887cc56f8-xkk4q\" (UID: \"d74849df-dcca-410f-93f2-7b84f72e6b59\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2b442-7887cc56f8-xkk4q"
Sep 09 15:57:08.007718 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:08.007591    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d74849df-dcca-410f-93f2-7b84f72e6b59-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a2b442-7887cc56f8-xkk4q\" (UID: \"d74849df-dcca-410f-93f2-7b84f72e6b59\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2b442-7887cc56f8-xkk4q"
Sep 09 15:57:08.007718 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:08.007614    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shqmj\" (UniqueName: \"kubernetes.io/projected/0452b747-ebc3-4807-87c3-927b2be91b08-kube-api-access-shqmj\") pod \"payload-processing-e2e-ipp-b-a2b442-9cbbd6765-4qnxr\" (UID: \"0452b747-ebc3-4807-87c3-927b2be91b08\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a2b442-9cbbd6765-4qnxr"
Sep 09 15:57:08.008264 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:08.008243    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0452b747-ebc3-4807-87c3-927b2be91b08-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a2b442-9cbbd6765-4qnxr\" (UID: \"0452b747-ebc3-4807-87c3-927b2be91b08\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a2b442-9cbbd6765-4qnxr"
Sep 09 15:57:08.008372 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:08.008352    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d74849df-dcca-410f-93f2-7b84f72e6b59-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a2b442-7887cc56f8-xkk4q\" (UID: \"d74849df-dcca-410f-93f2-7b84f72e6b59\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2b442-7887cc56f8-xkk4q"
Sep 09 15:57:08.015847 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:08.015783    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xndnj\" (UniqueName: \"kubernetes.io/projected/d74849df-dcca-410f-93f2-7b84f72e6b59-kube-api-access-xndnj\") pod \"payload-pre-processing-e2e-ipp-b-a2b442-7887cc56f8-xkk4q\" (UID: \"d74849df-dcca-410f-93f2-7b84f72e6b59\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2b442-7887cc56f8-xkk4q"
Sep 09 15:57:08.015847 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:08.015807    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shqmj\" (UniqueName: \"kubernetes.io/projected/0452b747-ebc3-4807-87c3-927b2be91b08-kube-api-access-shqmj\") pod \"payload-processing-e2e-ipp-b-a2b442-9cbbd6765-4qnxr\" (UID: \"0452b747-ebc3-4807-87c3-927b2be91b08\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a2b442-9cbbd6765-4qnxr"
Sep 09 15:57:08.077895 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:08.077855    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2b442-7887cc56f8-xkk4q"
Sep 09 15:57:08.122116 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:08.122080    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2b442-9cbbd6765-4qnxr"
Sep 09 15:57:08.349239 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:08.349208    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a2b442-7887cc56f8-xkk4q"]
Sep 09 15:57:08.350641 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:57:08.350614    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd74849df_dcca_410f_93f2_7b84f72e6b59.slice/crio-339b9ab0b20ba811301c287144073dbd7e92cb4e584ce603fbe58ccdb318496b WatchSource:0}: Error finding container 339b9ab0b20ba811301c287144073dbd7e92cb4e584ce603fbe58ccdb318496b: Status 404 returned error can't find the container with id 339b9ab0b20ba811301c287144073dbd7e92cb4e584ce603fbe58ccdb318496b
Sep 09 15:57:08.404023 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:08.403995    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a2b442-9cbbd6765-4qnxr"]
Sep 09 15:57:08.415885 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:57:08.415856    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0452b747_ebc3_4807_87c3_927b2be91b08.slice/crio-4c41503053ed0defdaa712bca3187484f6c1242761a224d8f7f292729c84b04c WatchSource:0}: Error finding container 4c41503053ed0defdaa712bca3187484f6c1242761a224d8f7f292729c84b04c: Status 404 returned error can't find the container with id 4c41503053ed0defdaa712bca3187484f6c1242761a224d8f7f292729c84b04c
Sep 09 15:57:08.502533 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:08.502493    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2b442-9cbbd6765-4qnxr" event={"ID":"0452b747-ebc3-4807-87c3-927b2be91b08","Type":"ContainerStarted","Data":"44d801ecaf950125f68f3284f5950c6c44b45556c00a40a6d76e56a6dea31285"}
Sep 09 15:57:08.502709 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:08.502565    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2b442-9cbbd6765-4qnxr"
Sep 09 15:57:08.502709 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:08.502588    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2b442-9cbbd6765-4qnxr" event={"ID":"0452b747-ebc3-4807-87c3-927b2be91b08","Type":"ContainerStarted","Data":"4c41503053ed0defdaa712bca3187484f6c1242761a224d8f7f292729c84b04c"}
Sep 09 15:57:08.504375 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:08.504345    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2b442-7887cc56f8-xkk4q" event={"ID":"d74849df-dcca-410f-93f2-7b84f72e6b59","Type":"ContainerStarted","Data":"636ee8cb01d014ce7625066ae3f2b64e530ceb692b7014a4fca0e37ce6c65f4c"}
Sep 09 15:57:08.504490 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:08.504386    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2b442-7887cc56f8-xkk4q" event={"ID":"d74849df-dcca-410f-93f2-7b84f72e6b59","Type":"ContainerStarted","Data":"339b9ab0b20ba811301c287144073dbd7e92cb4e584ce603fbe58ccdb318496b"}
Sep 09 15:57:08.504490 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:08.504411    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2b442-7887cc56f8-xkk4q"
Sep 09 15:57:08.521075 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:08.521024    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2b442-9cbbd6765-4qnxr" podStartSLOduration=1.5210053000000001 podStartE2EDuration="1.5210053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:57:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:57:08.520670159 +0000 UTC m=+2194.119564818" watchObservedRunningTime="2026-09-09 15:57:08.5210053 +0000 UTC m=+2194.119899945"
Sep 09 15:57:08.540544 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:08.540477    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2b442-7887cc56f8-xkk4q" podStartSLOduration=1.540455955 podStartE2EDuration="1.540455955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:57:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:57:08.538210009 +0000 UTC m=+2194.137104654" watchObservedRunningTime="2026-09-09 15:57:08.540455955 +0000 UTC m=+2194.139350603"
Sep 09 15:57:19.511744 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:19.511710    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2b442-7887cc56f8-xkk4q"
Sep 09 15:57:19.512230 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:19.511881    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2b442-9cbbd6765-4qnxr"
Sep 09 15:57:35.388181 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:35.388147    2553 scope.go:117] "RemoveContainer" containerID="1c609b61279a61678b642042b71f7b6a57aafd74b61db47c1712900b780ffaaf"
Sep 09 15:57:35.396654 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:35.396631    2553 scope.go:117] "RemoveContainer" containerID="8da24a2f0e920a067565a58c5e0a83da6c0105a5adb14b68214f6ac28005dc9b"
Sep 09 15:57:36.489941 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:57:36.489910    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a2b442-9cbbd6765-4qnxr_0452b747-ebc3-4807-87c3-927b2be91b08/payload-processing/0.log"
Sep 09 15:58:07.331662 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:07.331584    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-1218fe-8685d765b7-ph7rh_7f9365a6-cea0-4111-b12c-df9e0abd260e/payload-processing/0.log"
Sep 09 15:58:07.599326 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:07.599248    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a2b442-9cbbd6765-4qnxr_0452b747-ebc3-4807-87c3-927b2be91b08/payload-processing/0.log"
Sep 09 15:58:08.190920 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:08.190887    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"]
Sep 09 15:58:08.195951 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:08.195923    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"]
Sep 09 15:58:08.196109 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:08.196039    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"
Sep 09 15:58:08.198610 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:08.198584    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-c6ae87-openshift-default-dockercfg-lqjzm\""
Sep 09 15:58:08.364335 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:08.364299    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/723b80c1-6617-4647-88a9-66079b49488b-istiod-ca-cert\") pod \"e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj\" (UID: \"723b80c1-6617-4647-88a9-66079b49488b\") " pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"
Sep 09 15:58:08.364768 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:08.364417    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/723b80c1-6617-4647-88a9-66079b49488b-istio-token\") pod \"e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj\" (UID: \"723b80c1-6617-4647-88a9-66079b49488b\") " pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"
Sep 09 15:58:08.364768 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:08.364459    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/723b80c1-6617-4647-88a9-66079b49488b-workload-socket\") pod \"e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj\" (UID: \"723b80c1-6617-4647-88a9-66079b49488b\") " pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"
Sep 09 15:58:08.364768 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:08.364496    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/723b80c1-6617-4647-88a9-66079b49488b-istio-data\") pod \"e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj\" (UID: \"723b80c1-6617-4647-88a9-66079b49488b\") " pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"
Sep 09 15:58:08.364768 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:08.364545    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx6xg\" (UniqueName: \"kubernetes.io/projected/723b80c1-6617-4647-88a9-66079b49488b-kube-api-access-wx6xg\") pod \"e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj\" (UID: \"723b80c1-6617-4647-88a9-66079b49488b\") " pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"
Sep 09 15:58:08.364768 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:08.364578    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/723b80c1-6617-4647-88a9-66079b49488b-credential-socket\") pod \"e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj\" (UID: \"723b80c1-6617-4647-88a9-66079b49488b\") " pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"
Sep 09 15:58:08.364768 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:08.364622    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/723b80c1-6617-4647-88a9-66079b49488b-istio-podinfo\") pod \"e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj\" (UID: \"723b80c1-6617-4647-88a9-66079b49488b\") " pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"
Sep 09 15:58:08.364768 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:08.364643    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/723b80c1-6617-4647-88a9-66079b49488b-workload-certs\") pod \"e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj\" (UID: \"723b80c1-6617-4647-88a9-66079b49488b\") " pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"
Sep 09 15:58:08.364768 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:08.364664    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/723b80c1-6617-4647-88a9-66079b49488b-istio-envoy\") pod \"e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj\" (UID: \"723b80c1-6617-4647-88a9-66079b49488b\") " pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"
Sep 09 15:58:08.465988 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:08.465905    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/723b80c1-6617-4647-88a9-66079b49488b-istio-token\") pod \"e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj\" (UID: \"723b80c1-6617-4647-88a9-66079b49488b\") " pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"
Sep 09 15:58:08.465988 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:08.465941    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/723b80c1-6617-4647-88a9-66079b49488b-workload-socket\") pod \"e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj\" (UID: \"723b80c1-6617-4647-88a9-66079b49488b\") " pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"
Sep 09 15:58:08.465988 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:08.465974    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/723b80c1-6617-4647-88a9-66079b49488b-istio-data\") pod \"e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj\" (UID: \"723b80c1-6617-4647-88a9-66079b49488b\") " pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"
Sep 09 15:58:08.466215 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:08.466155    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx6xg\" (UniqueName: \"kubernetes.io/projected/723b80c1-6617-4647-88a9-66079b49488b-kube-api-access-wx6xg\") pod \"e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj\" (UID: \"723b80c1-6617-4647-88a9-66079b49488b\") " pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"
Sep 09 15:58:08.466215 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:08.466208    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/723b80c1-6617-4647-88a9-66079b49488b-credential-socket\") pod \"e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj\" (UID: \"723b80c1-6617-4647-88a9-66079b49488b\") " pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"
Sep 09 15:58:08.466325 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:08.466257    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/723b80c1-6617-4647-88a9-66079b49488b-istio-podinfo\") pod \"e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj\" (UID: \"723b80c1-6617-4647-88a9-66079b49488b\") " pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"
Sep 09 15:58:08.466325 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:08.466284    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/723b80c1-6617-4647-88a9-66079b49488b-workload-certs\") pod \"e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj\" (UID: \"723b80c1-6617-4647-88a9-66079b49488b\") " pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"
Sep 09 15:58:08.466325 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:08.466306    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/723b80c1-6617-4647-88a9-66079b49488b-istio-data\") pod \"e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj\" (UID: \"723b80c1-6617-4647-88a9-66079b49488b\") " pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"
Sep 09 15:58:08.466479 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:08.466335    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/723b80c1-6617-4647-88a9-66079b49488b-workload-socket\") pod \"e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj\" (UID: \"723b80c1-6617-4647-88a9-66079b49488b\") " pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"
Sep 09 15:58:08.466479 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:08.466362    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/723b80c1-6617-4647-88a9-66079b49488b-istio-envoy\") pod \"e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj\" (UID: \"723b80c1-6617-4647-88a9-66079b49488b\") " pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"
Sep 09 15:58:08.466593 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:08.466490    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/723b80c1-6617-4647-88a9-66079b49488b-istiod-ca-cert\") pod \"e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj\" (UID: \"723b80c1-6617-4647-88a9-66079b49488b\") " pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"
Sep 09 15:58:08.466593 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:08.466556    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/723b80c1-6617-4647-88a9-66079b49488b-credential-socket\") pod \"e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj\" (UID: \"723b80c1-6617-4647-88a9-66079b49488b\") " pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"
Sep 09 15:58:08.466675 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:08.466597    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/723b80c1-6617-4647-88a9-66079b49488b-workload-certs\") pod \"e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj\" (UID: \"723b80c1-6617-4647-88a9-66079b49488b\") " pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"
Sep 09 15:58:08.467100 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:08.467078    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/723b80c1-6617-4647-88a9-66079b49488b-istiod-ca-cert\") pod \"e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj\" (UID: \"723b80c1-6617-4647-88a9-66079b49488b\") " pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"
Sep 09 15:58:08.468597 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:08.468575    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/723b80c1-6617-4647-88a9-66079b49488b-istio-envoy\") pod \"e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj\" (UID: \"723b80c1-6617-4647-88a9-66079b49488b\") " pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"
Sep 09 15:58:08.468743 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:08.468712    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/723b80c1-6617-4647-88a9-66079b49488b-istio-podinfo\") pod \"e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj\" (UID: \"723b80c1-6617-4647-88a9-66079b49488b\") " pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"
Sep 09 15:58:08.474053 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:08.474027    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/723b80c1-6617-4647-88a9-66079b49488b-istio-token\") pod \"e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj\" (UID: \"723b80c1-6617-4647-88a9-66079b49488b\") " pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"
Sep 09 15:58:08.474200 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:08.474184    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx6xg\" (UniqueName: \"kubernetes.io/projected/723b80c1-6617-4647-88a9-66079b49488b-kube-api-access-wx6xg\") pod \"e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj\" (UID: \"723b80c1-6617-4647-88a9-66079b49488b\") " pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"
Sep 09 15:58:08.508158 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:08.508118    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"
Sep 09 15:58:08.784649 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:08.784620    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"]
Sep 09 15:58:08.786691 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:58:08.786658    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod723b80c1_6617_4647_88a9_66079b49488b.slice/crio-a05b988752d2915c9c0fa750dfc1867c03bec9dbe5122c3655e777e6821820c2 WatchSource:0}: Error finding container a05b988752d2915c9c0fa750dfc1867c03bec9dbe5122c3655e777e6821820c2: Status 404 returned error can't find the container with id a05b988752d2915c9c0fa750dfc1867c03bec9dbe5122c3655e777e6821820c2
Sep 09 15:58:08.789683 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:08.789638    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:58:08.789794 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:08.789757    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:58:08.789846 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:08.789803    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:58:08.809511 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:08.809476    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj" event={"ID":"723b80c1-6617-4647-88a9-66079b49488b","Type":"ContainerStarted","Data":"a05b988752d2915c9c0fa750dfc1867c03bec9dbe5122c3655e777e6821820c2"}
Sep 09 15:58:09.816302 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:09.816263    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj" event={"ID":"723b80c1-6617-4647-88a9-66079b49488b","Type":"ContainerStarted","Data":"77c539178b2484c7ca05ae75bc2662b5c3bd95d6e0a8661be9536e1365181652"}
Sep 09 15:58:09.838541 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:09.838486    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj" podStartSLOduration=1.838468689 podStartE2EDuration="1.838468689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:58:09.835986283 +0000 UTC m=+2255.434880927" watchObservedRunningTime="2026-09-09 15:58:09.838468689 +0000 UTC m=+2255.437363332"
Sep 09 15:58:10.508654 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:10.508614    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"
Sep 09 15:58:10.514574 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:10.514537    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"
Sep 09 15:58:10.821608 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:10.821522    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"
Sep 09 15:58:10.822515 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:10.822495    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"
Sep 09 15:58:32.936868 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:32.936831    2553 generic.go:356] "Generic (PLEG): container finished" podID="723b80c1-6617-4647-88a9-66079b49488b" containerID="77c539178b2484c7ca05ae75bc2662b5c3bd95d6e0a8661be9536e1365181652" exitCode=0
Sep 09 15:58:32.937306 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:32.936912    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj" event={"ID":"723b80c1-6617-4647-88a9-66079b49488b","Type":"ContainerDied","Data":"77c539178b2484c7ca05ae75bc2662b5c3bd95d6e0a8661be9536e1365181652"}
Sep 09 15:58:32.937352 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:32.937332    2553 scope.go:117] "RemoveContainer" containerID="77c539178b2484c7ca05ae75bc2662b5c3bd95d6e0a8661be9536e1365181652"
Sep 09 15:58:32.938647 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:32.938612    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:58:32.938718 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:32.938702    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:58:32.938782 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:32.938765    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 15:58:33.953024 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:33.952986    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj" event={"ID":"723b80c1-6617-4647-88a9-66079b49488b","Type":"ContainerStarted","Data":"b983464a5e1df20b921354bbd6e9200624a82659c8c064dc8431b57ada7c70f4"}
Sep 09 15:58:34.509161 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:34.509122    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"
Sep 09 15:58:34.509395 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:34.509373    2553 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Sep 09 15:58:34.509450 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:34.509421    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj" podUID="723b80c1-6617-4647-88a9-66079b49488b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Sep 09 15:58:35.428785 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:35.428756    2553 scope.go:117] "RemoveContainer" containerID="9393a38f28d43b4cb22dd216f0662e2431bbb7dd953ab902ad7679e3d271566d"
Sep 09 15:58:35.450939 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:35.450908    2553 scope.go:117] "RemoveContainer" containerID="b3db36d0d490828061ef133b2ff954b8a091b6acddf64fe70f4bca3742d076e7"
Sep 09 15:58:35.509292 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:35.509263    2553 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Sep 09 15:58:35.509418 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:35.509326    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj" podUID="723b80c1-6617-4647-88a9-66079b49488b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Sep 09 15:58:36.509056 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:36.509019    2553 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Sep 09 15:58:36.509502 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:36.509092    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj" podUID="723b80c1-6617-4647-88a9-66079b49488b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Sep 09 15:58:37.509138 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:37.509100    2553 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Sep 09 15:58:37.509544 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:37.509171    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj" podUID="723b80c1-6617-4647-88a9-66079b49488b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Sep 09 15:58:37.670245 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:37.670211    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c6ae87-gj5pk"]
Sep 09 15:58:37.673520 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:37.673501    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c6ae87-gj5pk"
Sep 09 15:58:37.676176 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:37.676157    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2zlsf\""
Sep 09 15:58:37.676273 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:37.676192    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 15:58:37.680159 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:37.680130    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c6ae87-gj5pk"]
Sep 09 15:58:37.760824 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:37.756298    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2d00007-e143-4426-86a7-309448ef8f90-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-c6ae87-gj5pk\" (UID: \"e2d00007-e143-4426-86a7-309448ef8f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c6ae87-gj5pk"
Sep 09 15:58:37.857396 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:37.857338    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2d00007-e143-4426-86a7-309448ef8f90-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-c6ae87-gj5pk\" (UID: \"e2d00007-e143-4426-86a7-309448ef8f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c6ae87-gj5pk"
Sep 09 15:58:37.858298 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:37.858264    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2d00007-e143-4426-86a7-309448ef8f90-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-c6ae87-gj5pk\" (UID: \"e2d00007-e143-4426-86a7-309448ef8f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c6ae87-gj5pk"
Sep 09 15:58:37.986759 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:37.986706    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c6ae87-gj5pk"
Sep 09 15:58:38.265036 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:38.264952    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c6ae87-gj5pk"]
Sep 09 15:58:38.267576 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:58:38.267543    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2d00007_e143_4426_86a7_309448ef8f90.slice/crio-7d6a132241684a2f8a178b6dd4dcc31015c50eeb4ca68af687b56cd1296cf4b0 WatchSource:0}: Error finding container 7d6a132241684a2f8a178b6dd4dcc31015c50eeb4ca68af687b56cd1296cf4b0: Status 404 returned error can't find the container with id 7d6a132241684a2f8a178b6dd4dcc31015c50eeb4ca68af687b56cd1296cf4b0
Sep 09 15:58:38.508669 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:38.508633    2553 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Sep 09 15:58:38.508859 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:38.508691    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj" podUID="723b80c1-6617-4647-88a9-66079b49488b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Sep 09 15:58:38.990017 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:38.989980    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c6ae87-gj5pk" event={"ID":"e2d00007-e143-4426-86a7-309448ef8f90","Type":"ContainerStarted","Data":"187bb6e3c90f80ce8c1c6132a64aad0bf54bba97831fd698fc084f88321367a3"}
Sep 09 15:58:38.990397 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:38.990044    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c6ae87-gj5pk" event={"ID":"e2d00007-e143-4426-86a7-309448ef8f90","Type":"ContainerStarted","Data":"7d6a132241684a2f8a178b6dd4dcc31015c50eeb4ca68af687b56cd1296cf4b0"}
Sep 09 15:58:39.008899 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:39.008844    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c6ae87-gj5pk" podStartSLOduration=2.00882768 podStartE2EDuration="2.00882768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:58:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:58:39.007579415 +0000 UTC m=+2284.606474058" watchObservedRunningTime="2026-09-09 15:58:39.00882768 +0000 UTC m=+2284.607722325"
Sep 09 15:58:39.553276 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:39.553095    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"
Sep 09 15:58:39.553504 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:39.553466    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"
Sep 09 15:58:39.554442 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:39.554423    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"
Sep 09 15:58:39.996864 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:39.996831    2553 generic.go:356] "Generic (PLEG): container finished" podID="e2d00007-e143-4426-86a7-309448ef8f90" containerID="187bb6e3c90f80ce8c1c6132a64aad0bf54bba97831fd698fc084f88321367a3" exitCode=0
Sep 09 15:58:39.997252 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:39.996917    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c6ae87-gj5pk" event={"ID":"e2d00007-e143-4426-86a7-309448ef8f90","Type":"ContainerDied","Data":"187bb6e3c90f80ce8c1c6132a64aad0bf54bba97831fd698fc084f88321367a3"}
Sep 09 15:58:41.138331 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:41.138300    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c6ae87-gj5pk"
Sep 09 15:58:41.294135 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:41.294017    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2d00007-e143-4426-86a7-309448ef8f90-maas-api-service-ca\") pod \"e2d00007-e143-4426-86a7-309448ef8f90\" (UID: \"e2d00007-e143-4426-86a7-309448ef8f90\") "
Sep 09 15:58:41.294441 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:41.294413    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2d00007-e143-4426-86a7-309448ef8f90-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e2d00007-e143-4426-86a7-309448ef8f90" (UID: "e2d00007-e143-4426-86a7-309448ef8f90"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:58:41.395863 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:41.395721    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2d00007-e143-4426-86a7-309448ef8f90-maas-api-service-ca\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:58:42.009766 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:42.009718    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c6ae87-gj5pk"
Sep 09 15:58:42.009766 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:42.009757    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c6ae87-gj5pk" event={"ID":"e2d00007-e143-4426-86a7-309448ef8f90","Type":"ContainerDied","Data":"7d6a132241684a2f8a178b6dd4dcc31015c50eeb4ca68af687b56cd1296cf4b0"}
Sep 09 15:58:42.009977 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:42.009807    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d6a132241684a2f8a178b6dd4dcc31015c50eeb4ca68af687b56cd1296cf4b0"
Sep 09 15:58:43.870263 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:43.870220    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c6ae87-gj5pk"]
Sep 09 15:58:43.872603 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:43.872579    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c6ae87-gj5pk"]
Sep 09 15:58:44.962632 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:44.962592    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2d00007-e143-4426-86a7-309448ef8f90" path="/var/lib/kubelet/pods/e2d00007-e143-4426-86a7-309448ef8f90/volumes"
Sep 09 15:58:50.262701 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:50.262668    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"]
Sep 09 15:58:50.263160 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:50.262926    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj" podUID="723b80c1-6617-4647-88a9-66079b49488b" containerName="istio-proxy" containerID="cri-o://b983464a5e1df20b921354bbd6e9200624a82659c8c064dc8431b57ada7c70f4" gracePeriod=30
Sep 09 15:58:54.328456 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:54.328423    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1218fe-gmtrr"]
Sep 09 15:58:54.328866 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:54.328819    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2d00007-e143-4426-86a7-309448ef8f90" containerName="revoke-keys"
Sep 09 15:58:54.328866 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:54.328833    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2d00007-e143-4426-86a7-309448ef8f90" containerName="revoke-keys"
Sep 09 15:58:54.328936 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:54.328902    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2d00007-e143-4426-86a7-309448ef8f90" containerName="revoke-keys"
Sep 09 15:58:54.331981 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:54.331965    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1218fe-gmtrr"
Sep 09 15:58:54.334538 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:54.334514    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 15:58:54.334685 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:54.334553    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2zlsf\""
Sep 09 15:58:54.339101 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:54.339023    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1218fe-gmtrr"]
Sep 09 15:58:54.410098 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:54.410052    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb75313e-7153-437d-bcde-87292c5fd3de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1218fe-gmtrr\" (UID: \"fb75313e-7153-437d-bcde-87292c5fd3de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1218fe-gmtrr"
Sep 09 15:58:54.511644 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:54.511599    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb75313e-7153-437d-bcde-87292c5fd3de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1218fe-gmtrr\" (UID: \"fb75313e-7153-437d-bcde-87292c5fd3de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1218fe-gmtrr"
Sep 09 15:58:54.512263 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:54.512241    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb75313e-7153-437d-bcde-87292c5fd3de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1218fe-gmtrr\" (UID: \"fb75313e-7153-437d-bcde-87292c5fd3de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1218fe-gmtrr"
Sep 09 15:58:54.555824 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:54.555786    2553 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 15:58:54.555981 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:54.555870    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj" podUID="723b80c1-6617-4647-88a9-66079b49488b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 15:58:54.645817 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:54.645701    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1218fe-gmtrr"
Sep 09 15:58:54.918945 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:54.918892    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1218fe-gmtrr"]
Sep 09 15:58:54.921419 ip-10-0-129-170 kubenswrapper[2553]: W0909 15:58:54.921397    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb75313e_7153_437d_bcde_87292c5fd3de.slice/crio-efebc93d8e9f1bf80656a200101cbf238522860dc2de398e5f5636cb67dc540e WatchSource:0}: Error finding container efebc93d8e9f1bf80656a200101cbf238522860dc2de398e5f5636cb67dc540e: Status 404 returned error can't find the container with id efebc93d8e9f1bf80656a200101cbf238522860dc2de398e5f5636cb67dc540e
Sep 09 15:58:55.076495 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:55.076452    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1218fe-gmtrr" event={"ID":"fb75313e-7153-437d-bcde-87292c5fd3de","Type":"ContainerStarted","Data":"efebc93d8e9f1bf80656a200101cbf238522860dc2de398e5f5636cb67dc540e"}
Sep 09 15:58:55.506087 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:55.506058    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"
Sep 09 15:58:55.521678 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:55.521641    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/723b80c1-6617-4647-88a9-66079b49488b-istiod-ca-cert\") pod \"723b80c1-6617-4647-88a9-66079b49488b\" (UID: \"723b80c1-6617-4647-88a9-66079b49488b\") "
Sep 09 15:58:55.521678 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:55.521683    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/723b80c1-6617-4647-88a9-66079b49488b-workload-socket\") pod \"723b80c1-6617-4647-88a9-66079b49488b\" (UID: \"723b80c1-6617-4647-88a9-66079b49488b\") "
Sep 09 15:58:55.521932 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:55.521716    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/723b80c1-6617-4647-88a9-66079b49488b-istio-token\") pod \"723b80c1-6617-4647-88a9-66079b49488b\" (UID: \"723b80c1-6617-4647-88a9-66079b49488b\") "
Sep 09 15:58:55.521932 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:55.521763    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/723b80c1-6617-4647-88a9-66079b49488b-credential-socket\") pod \"723b80c1-6617-4647-88a9-66079b49488b\" (UID: \"723b80c1-6617-4647-88a9-66079b49488b\") "
Sep 09 15:58:55.521932 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:55.521785    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/723b80c1-6617-4647-88a9-66079b49488b-istio-envoy\") pod \"723b80c1-6617-4647-88a9-66079b49488b\" (UID: \"723b80c1-6617-4647-88a9-66079b49488b\") "
Sep 09 15:58:55.521932 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:55.521817    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/723b80c1-6617-4647-88a9-66079b49488b-istio-data\") pod \"723b80c1-6617-4647-88a9-66079b49488b\" (UID: \"723b80c1-6617-4647-88a9-66079b49488b\") "
Sep 09 15:58:55.521932 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:55.521837    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wx6xg\" (UniqueName: \"kubernetes.io/projected/723b80c1-6617-4647-88a9-66079b49488b-kube-api-access-wx6xg\") pod \"723b80c1-6617-4647-88a9-66079b49488b\" (UID: \"723b80c1-6617-4647-88a9-66079b49488b\") "
Sep 09 15:58:55.521932 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:55.521871    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/723b80c1-6617-4647-88a9-66079b49488b-workload-certs\") pod \"723b80c1-6617-4647-88a9-66079b49488b\" (UID: \"723b80c1-6617-4647-88a9-66079b49488b\") "
Sep 09 15:58:55.521932 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:55.521915    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/723b80c1-6617-4647-88a9-66079b49488b-istio-podinfo\") pod \"723b80c1-6617-4647-88a9-66079b49488b\" (UID: \"723b80c1-6617-4647-88a9-66079b49488b\") "
Sep 09 15:58:55.522304 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:55.521992    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723b80c1-6617-4647-88a9-66079b49488b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "723b80c1-6617-4647-88a9-66079b49488b" (UID: "723b80c1-6617-4647-88a9-66079b49488b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:58:55.522304 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:55.522139    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723b80c1-6617-4647-88a9-66079b49488b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "723b80c1-6617-4647-88a9-66079b49488b" (UID: "723b80c1-6617-4647-88a9-66079b49488b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:58:55.522304 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:55.522156    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723b80c1-6617-4647-88a9-66079b49488b-istio-data" (OuterVolumeSpecName: "istio-data") pod "723b80c1-6617-4647-88a9-66079b49488b" (UID: "723b80c1-6617-4647-88a9-66079b49488b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:58:55.522304 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:55.522163    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/723b80c1-6617-4647-88a9-66079b49488b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "723b80c1-6617-4647-88a9-66079b49488b" (UID: "723b80c1-6617-4647-88a9-66079b49488b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:58:55.522304 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:55.522223    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723b80c1-6617-4647-88a9-66079b49488b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "723b80c1-6617-4647-88a9-66079b49488b" (UID: "723b80c1-6617-4647-88a9-66079b49488b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:58:55.522564 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:55.522335    2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/723b80c1-6617-4647-88a9-66079b49488b-credential-socket\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:58:55.522564 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:55.522354    2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/723b80c1-6617-4647-88a9-66079b49488b-istio-data\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:58:55.522564 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:55.522369    2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/723b80c1-6617-4647-88a9-66079b49488b-workload-certs\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:58:55.522564 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:55.522381    2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/723b80c1-6617-4647-88a9-66079b49488b-istiod-ca-cert\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:58:55.522564 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:55.522394    2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/723b80c1-6617-4647-88a9-66079b49488b-workload-socket\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:58:55.524140 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:55.524113    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/723b80c1-6617-4647-88a9-66079b49488b-kube-api-access-wx6xg" (OuterVolumeSpecName: "kube-api-access-wx6xg") pod "723b80c1-6617-4647-88a9-66079b49488b" (UID: "723b80c1-6617-4647-88a9-66079b49488b"). InnerVolumeSpecName "kube-api-access-wx6xg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:58:55.524253 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:55.524155    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723b80c1-6617-4647-88a9-66079b49488b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "723b80c1-6617-4647-88a9-66079b49488b" (UID: "723b80c1-6617-4647-88a9-66079b49488b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:58:55.524253 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:55.524173    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/723b80c1-6617-4647-88a9-66079b49488b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "723b80c1-6617-4647-88a9-66079b49488b" (UID: "723b80c1-6617-4647-88a9-66079b49488b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 15:58:55.524253 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:55.524184    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/723b80c1-6617-4647-88a9-66079b49488b-istio-token" (OuterVolumeSpecName: "istio-token") pod "723b80c1-6617-4647-88a9-66079b49488b" (UID: "723b80c1-6617-4647-88a9-66079b49488b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:58:55.623391 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:55.623336    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx6xg\" (UniqueName: \"kubernetes.io/projected/723b80c1-6617-4647-88a9-66079b49488b-kube-api-access-wx6xg\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:58:55.623391 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:55.623383    2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/723b80c1-6617-4647-88a9-66079b49488b-istio-podinfo\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:58:55.623391 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:55.623393    2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/723b80c1-6617-4647-88a9-66079b49488b-istio-token\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:58:55.623391 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:55.623401    2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/723b80c1-6617-4647-88a9-66079b49488b-istio-envoy\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:58:56.083093 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:56.083050    2553 generic.go:356] "Generic (PLEG): container finished" podID="fb75313e-7153-437d-bcde-87292c5fd3de" containerID="0bfe74d55edfe5fb31fae0423eec8d98433e4b834d66a8fa4092f83eb8da9d2d" exitCode=0
Sep 09 15:58:56.083251 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:56.083131    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1218fe-gmtrr" event={"ID":"fb75313e-7153-437d-bcde-87292c5fd3de","Type":"ContainerDied","Data":"0bfe74d55edfe5fb31fae0423eec8d98433e4b834d66a8fa4092f83eb8da9d2d"}
Sep 09 15:58:56.085017 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:56.084992    2553 generic.go:356] "Generic (PLEG): container finished" podID="723b80c1-6617-4647-88a9-66079b49488b" containerID="b983464a5e1df20b921354bbd6e9200624a82659c8c064dc8431b57ada7c70f4" exitCode=0
Sep 09 15:58:56.085137 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:56.085031    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj" event={"ID":"723b80c1-6617-4647-88a9-66079b49488b","Type":"ContainerDied","Data":"b983464a5e1df20b921354bbd6e9200624a82659c8c064dc8431b57ada7c70f4"}
Sep 09 15:58:56.085137 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:56.085041    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"
Sep 09 15:58:56.085137 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:56.085079    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj" event={"ID":"723b80c1-6617-4647-88a9-66079b49488b","Type":"ContainerDied","Data":"a05b988752d2915c9c0fa750dfc1867c03bec9dbe5122c3655e777e6821820c2"}
Sep 09 15:58:56.085137 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:56.085115    2553 scope.go:117] "RemoveContainer" containerID="b983464a5e1df20b921354bbd6e9200624a82659c8c064dc8431b57ada7c70f4"
Sep 09 15:58:56.094084 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:56.094066    2553 scope.go:117] "RemoveContainer" containerID="77c539178b2484c7ca05ae75bc2662b5c3bd95d6e0a8661be9536e1365181652"
Sep 09 15:58:56.102463 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:56.102441    2553 scope.go:117] "RemoveContainer" containerID="b983464a5e1df20b921354bbd6e9200624a82659c8c064dc8431b57ada7c70f4"
Sep 09 15:58:56.102723 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:58:56.102701    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b983464a5e1df20b921354bbd6e9200624a82659c8c064dc8431b57ada7c70f4\": container with ID starting with b983464a5e1df20b921354bbd6e9200624a82659c8c064dc8431b57ada7c70f4 not found: ID does not exist" containerID="b983464a5e1df20b921354bbd6e9200624a82659c8c064dc8431b57ada7c70f4"
Sep 09 15:58:56.102924 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:56.102753    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b983464a5e1df20b921354bbd6e9200624a82659c8c064dc8431b57ada7c70f4"} err="failed to get container status \"b983464a5e1df20b921354bbd6e9200624a82659c8c064dc8431b57ada7c70f4\": rpc error: code = NotFound desc = could not find container \"b983464a5e1df20b921354bbd6e9200624a82659c8c064dc8431b57ada7c70f4\": container with ID starting with b983464a5e1df20b921354bbd6e9200624a82659c8c064dc8431b57ada7c70f4 not found: ID does not exist"
Sep 09 15:58:56.102924 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:56.102770    2553 scope.go:117] "RemoveContainer" containerID="77c539178b2484c7ca05ae75bc2662b5c3bd95d6e0a8661be9536e1365181652"
Sep 09 15:58:56.103050 ip-10-0-129-170 kubenswrapper[2553]: E0909 15:58:56.103006    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77c539178b2484c7ca05ae75bc2662b5c3bd95d6e0a8661be9536e1365181652\": container with ID starting with 77c539178b2484c7ca05ae75bc2662b5c3bd95d6e0a8661be9536e1365181652 not found: ID does not exist" containerID="77c539178b2484c7ca05ae75bc2662b5c3bd95d6e0a8661be9536e1365181652"
Sep 09 15:58:56.103050 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:56.103029    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77c539178b2484c7ca05ae75bc2662b5c3bd95d6e0a8661be9536e1365181652"} err="failed to get container status \"77c539178b2484c7ca05ae75bc2662b5c3bd95d6e0a8661be9536e1365181652\": rpc error: code = NotFound desc = could not find container \"77c539178b2484c7ca05ae75bc2662b5c3bd95d6e0a8661be9536e1365181652\": container with ID starting with 77c539178b2484c7ca05ae75bc2662b5c3bd95d6e0a8661be9536e1365181652 not found: ID does not exist"
Sep 09 15:58:56.115397 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:56.115367    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"]
Sep 09 15:58:56.120841 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:56.120811    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-c6ae87-openshift-default-c8d5c5496-68bjj"]
Sep 09 15:58:56.963377 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:56.963340    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="723b80c1-6617-4647-88a9-66079b49488b" path="/var/lib/kubelet/pods/723b80c1-6617-4647-88a9-66079b49488b/volumes"
Sep 09 15:58:57.229781 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:57.229709    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1218fe-gmtrr"
Sep 09 15:58:57.237293 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:57.237268    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb75313e-7153-437d-bcde-87292c5fd3de-maas-api-service-ca\") pod \"fb75313e-7153-437d-bcde-87292c5fd3de\" (UID: \"fb75313e-7153-437d-bcde-87292c5fd3de\") "
Sep 09 15:58:57.237596 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:57.237574    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb75313e-7153-437d-bcde-87292c5fd3de-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fb75313e-7153-437d-bcde-87292c5fd3de" (UID: "fb75313e-7153-437d-bcde-87292c5fd3de"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:58:57.338543 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:57.338499    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb75313e-7153-437d-bcde-87292c5fd3de-maas-api-service-ca\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:58:58.097679 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:58.097646    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1218fe-gmtrr"
Sep 09 15:58:58.098079 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:58.097654    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1218fe-gmtrr" event={"ID":"fb75313e-7153-437d-bcde-87292c5fd3de","Type":"ContainerDied","Data":"efebc93d8e9f1bf80656a200101cbf238522860dc2de398e5f5636cb67dc540e"}
Sep 09 15:58:58.098079 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:58:58.097787    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efebc93d8e9f1bf80656a200101cbf238522860dc2de398e5f5636cb67dc540e"
Sep 09 15:59:09.910572 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:59:09.910533    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1218fe-8685d765b7-ph7rh"]
Sep 09 15:59:09.911160 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:59:09.910832    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-1218fe-8685d765b7-ph7rh" podUID="7f9365a6-cea0-4111-b12c-df9e0abd260e" containerName="payload-processing" containerID="cri-o://af5229a4d1cf66a38b80722051c200ebcbde5426c72ed3c5a5792e027cb3642c" gracePeriod=30
Sep 09 15:59:09.919264 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:59:09.919239    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1218fe-cc75f9b58-v28rq"]
Sep 09 15:59:09.919625 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:59:09.919593    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1218fe-cc75f9b58-v28rq" podUID="d3bfd8e3-2dc5-45e5-83c8-af854c0354c5" containerName="payload-pre-processing" containerID="cri-o://dbb21718336261f8d834513915645c1e8f363db6405eecbd93c0e4eae3889133" gracePeriod=30
Sep 09 15:59:10.123073 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:59:10.123030    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1218fe-gmtrr"]
Sep 09 15:59:10.129462 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:59:10.129437    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1218fe-gmtrr"]
Sep 09 15:59:10.160193 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:59:10.160159    2553 generic.go:356] "Generic (PLEG): container finished" podID="7f9365a6-cea0-4111-b12c-df9e0abd260e" containerID="af5229a4d1cf66a38b80722051c200ebcbde5426c72ed3c5a5792e027cb3642c" exitCode=0
Sep 09 15:59:10.160386 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:59:10.160239    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1218fe-8685d765b7-ph7rh" event={"ID":"7f9365a6-cea0-4111-b12c-df9e0abd260e","Type":"ContainerDied","Data":"af5229a4d1cf66a38b80722051c200ebcbde5426c72ed3c5a5792e027cb3642c"}
Sep 09 15:59:10.162227 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:59:10.162145    2553 generic.go:356] "Generic (PLEG): container finished" podID="d3bfd8e3-2dc5-45e5-83c8-af854c0354c5" containerID="dbb21718336261f8d834513915645c1e8f363db6405eecbd93c0e4eae3889133" exitCode=0
Sep 09 15:59:10.162336 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:59:10.162228    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1218fe-cc75f9b58-v28rq" event={"ID":"d3bfd8e3-2dc5-45e5-83c8-af854c0354c5","Type":"ContainerDied","Data":"dbb21718336261f8d834513915645c1e8f363db6405eecbd93c0e4eae3889133"}
Sep 09 15:59:10.210659 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:59:10.210630    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1218fe-8685d765b7-ph7rh"
Sep 09 15:59:10.213220 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:59:10.213198    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1218fe-cc75f9b58-v28rq"
Sep 09 15:59:10.255624 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:59:10.255582    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfhq4\" (UniqueName: \"kubernetes.io/projected/d3bfd8e3-2dc5-45e5-83c8-af854c0354c5-kube-api-access-lfhq4\") pod \"d3bfd8e3-2dc5-45e5-83c8-af854c0354c5\" (UID: \"d3bfd8e3-2dc5-45e5-83c8-af854c0354c5\") "
Sep 09 15:59:10.255841 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:59:10.255680    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3bfd8e3-2dc5-45e5-83c8-af854c0354c5-plugins-config-volume\") pod \"d3bfd8e3-2dc5-45e5-83c8-af854c0354c5\" (UID: \"d3bfd8e3-2dc5-45e5-83c8-af854c0354c5\") "
Sep 09 15:59:10.255841 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:59:10.255715    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f9365a6-cea0-4111-b12c-df9e0abd260e-plugins-config-volume\") pod \"7f9365a6-cea0-4111-b12c-df9e0abd260e\" (UID: \"7f9365a6-cea0-4111-b12c-df9e0abd260e\") "
Sep 09 15:59:10.255841 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:59:10.255765    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkct4\" (UniqueName: \"kubernetes.io/projected/7f9365a6-cea0-4111-b12c-df9e0abd260e-kube-api-access-zkct4\") pod \"7f9365a6-cea0-4111-b12c-df9e0abd260e\" (UID: \"7f9365a6-cea0-4111-b12c-df9e0abd260e\") "
Sep 09 15:59:10.256094 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:59:10.256064    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3bfd8e3-2dc5-45e5-83c8-af854c0354c5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d3bfd8e3-2dc5-45e5-83c8-af854c0354c5" (UID: "d3bfd8e3-2dc5-45e5-83c8-af854c0354c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:59:10.256094 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:59:10.256084    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f9365a6-cea0-4111-b12c-df9e0abd260e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7f9365a6-cea0-4111-b12c-df9e0abd260e" (UID: "7f9365a6-cea0-4111-b12c-df9e0abd260e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:59:10.257922 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:59:10.257892    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f9365a6-cea0-4111-b12c-df9e0abd260e-kube-api-access-zkct4" (OuterVolumeSpecName: "kube-api-access-zkct4") pod "7f9365a6-cea0-4111-b12c-df9e0abd260e" (UID: "7f9365a6-cea0-4111-b12c-df9e0abd260e"). InnerVolumeSpecName "kube-api-access-zkct4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:59:10.258013 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:59:10.257918    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3bfd8e3-2dc5-45e5-83c8-af854c0354c5-kube-api-access-lfhq4" (OuterVolumeSpecName: "kube-api-access-lfhq4") pod "d3bfd8e3-2dc5-45e5-83c8-af854c0354c5" (UID: "d3bfd8e3-2dc5-45e5-83c8-af854c0354c5"). InnerVolumeSpecName "kube-api-access-lfhq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:59:10.356609 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:59:10.356570    2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f9365a6-cea0-4111-b12c-df9e0abd260e-plugins-config-volume\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:59:10.356609 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:59:10.356604    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkct4\" (UniqueName: \"kubernetes.io/projected/7f9365a6-cea0-4111-b12c-df9e0abd260e-kube-api-access-zkct4\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:59:10.356609 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:59:10.356614    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfhq4\" (UniqueName: \"kubernetes.io/projected/d3bfd8e3-2dc5-45e5-83c8-af854c0354c5-kube-api-access-lfhq4\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:59:10.356908 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:59:10.356623    2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3bfd8e3-2dc5-45e5-83c8-af854c0354c5-plugins-config-volume\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 15:59:10.962720 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:59:10.962685    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb75313e-7153-437d-bcde-87292c5fd3de" path="/var/lib/kubelet/pods/fb75313e-7153-437d-bcde-87292c5fd3de/volumes"
Sep 09 15:59:11.168473 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:59:11.168445    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1218fe-8685d765b7-ph7rh"
Sep 09 15:59:11.168667 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:59:11.168470    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1218fe-8685d765b7-ph7rh" event={"ID":"7f9365a6-cea0-4111-b12c-df9e0abd260e","Type":"ContainerDied","Data":"f7fe9b3f30ec273f7d3f17ba3e2cb8da39c66d8834393173905889f1a7d2687d"}
Sep 09 15:59:11.168667 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:59:11.168536    2553 scope.go:117] "RemoveContainer" containerID="af5229a4d1cf66a38b80722051c200ebcbde5426c72ed3c5a5792e027cb3642c"
Sep 09 15:59:11.169978 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:59:11.169926    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1218fe-cc75f9b58-v28rq" event={"ID":"d3bfd8e3-2dc5-45e5-83c8-af854c0354c5","Type":"ContainerDied","Data":"df6198cf538d8d47cddfe70d0d945f3f044ebf4d6a485a3e62d490e2d93774b5"}
Sep 09 15:59:11.169978 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:59:11.169948    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1218fe-cc75f9b58-v28rq"
Sep 09 15:59:11.177672 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:59:11.177651    2553 scope.go:117] "RemoveContainer" containerID="dbb21718336261f8d834513915645c1e8f363db6405eecbd93c0e4eae3889133"
Sep 09 15:59:11.188037 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:59:11.188004    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1218fe-cc75f9b58-v28rq"]
Sep 09 15:59:11.194187 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:59:11.194164    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1218fe-cc75f9b58-v28rq"]
Sep 09 15:59:11.204186 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:59:11.204162    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1218fe-8685d765b7-ph7rh"]
Sep 09 15:59:11.208072 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:59:11.208048    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1218fe-8685d765b7-ph7rh"]
Sep 09 15:59:12.963018 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:59:12.962985    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f9365a6-cea0-4111-b12c-df9e0abd260e" path="/var/lib/kubelet/pods/7f9365a6-cea0-4111-b12c-df9e0abd260e/volumes"
Sep 09 15:59:12.963545 ip-10-0-129-170 kubenswrapper[2553]: I0909 15:59:12.963522    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3bfd8e3-2dc5-45e5-83c8-af854c0354c5" path="/var/lib/kubelet/pods/d3bfd8e3-2dc5-45e5-83c8-af854c0354c5/volumes"
Sep 09 16:00:00.137654 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:00.137616    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816160-mpgjw"]
Sep 09 16:00:00.138347 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:00.138206    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="723b80c1-6617-4647-88a9-66079b49488b" containerName="istio-proxy"
Sep 09 16:00:00.138347 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:00.138244    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="723b80c1-6617-4647-88a9-66079b49488b" containerName="istio-proxy"
Sep 09 16:00:00.138347 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:00.138258    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb75313e-7153-437d-bcde-87292c5fd3de" containerName="revoke-keys"
Sep 09 16:00:00.138347 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:00.138267    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb75313e-7153-437d-bcde-87292c5fd3de" containerName="revoke-keys"
Sep 09 16:00:00.138347 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:00.138277    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3bfd8e3-2dc5-45e5-83c8-af854c0354c5" containerName="payload-pre-processing"
Sep 09 16:00:00.138347 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:00.138287    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3bfd8e3-2dc5-45e5-83c8-af854c0354c5" containerName="payload-pre-processing"
Sep 09 16:00:00.138347 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:00.138322    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="723b80c1-6617-4647-88a9-66079b49488b" containerName="istio-proxy"
Sep 09 16:00:00.138347 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:00.138330    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="723b80c1-6617-4647-88a9-66079b49488b" containerName="istio-proxy"
Sep 09 16:00:00.138887 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:00.138362    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f9365a6-cea0-4111-b12c-df9e0abd260e" containerName="payload-processing"
Sep 09 16:00:00.138887 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:00.138370    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f9365a6-cea0-4111-b12c-df9e0abd260e" containerName="payload-processing"
Sep 09 16:00:00.138887 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:00.138472    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="723b80c1-6617-4647-88a9-66079b49488b" containerName="istio-proxy"
Sep 09 16:00:00.138887 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:00.138485    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb75313e-7153-437d-bcde-87292c5fd3de" containerName="revoke-keys"
Sep 09 16:00:00.138887 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:00.138495    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f9365a6-cea0-4111-b12c-df9e0abd260e" containerName="payload-processing"
Sep 09 16:00:00.138887 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:00.138505    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3bfd8e3-2dc5-45e5-83c8-af854c0354c5" containerName="payload-pre-processing"
Sep 09 16:00:00.141645 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:00.141623    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-482dce-29816160-2nzdm"]
Sep 09 16:00:00.141784 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:00.141756    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816160-mpgjw"
Sep 09 16:00:00.142081 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:00.142058    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="723b80c1-6617-4647-88a9-66079b49488b" containerName="istio-proxy"
Sep 09 16:00:00.144305 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:00.144282    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2zlsf\""
Sep 09 16:00:00.144833 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:00.144817    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a2b442-29816160-xs9sc"]
Sep 09 16:00:00.144933 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:00.144918    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-482dce-29816160-2nzdm"
Sep 09 16:00:00.147944 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:00.147922    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a2b442-29816160-xs9sc"
Sep 09 16:00:00.151147 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:00.150698    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816160-mpgjw"]
Sep 09 16:00:00.153380 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:00.153356    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-482dce-29816160-2nzdm"]
Sep 09 16:00:00.155027 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:00.155002    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816160-mpgjw"
Sep 09 16:00:00.155780 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:00.155754    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a2b442-29816160-xs9sc"]
Sep 09 16:00:00.165276 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:00.165247    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-482dce-29816160-2nzdm"
Sep 09 16:00:00.203448 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:00.203420    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a2b442-29816160-xs9sc"
Sep 09 16:00:00.436496 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:00.436469    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816160-mpgjw"]
Sep 09 16:00:00.438654 ip-10-0-129-170 kubenswrapper[2553]: W0909 16:00:00.438626    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25e8a0ab_9725_47d2_9c06_5cb0db65f98e.slice/crio-e8c412eac4ba9286d804aac23630bedc5800b550c0a0ca325975c2790bde4c59 WatchSource:0}: Error finding container e8c412eac4ba9286d804aac23630bedc5800b550c0a0ca325975c2790bde4c59: Status 404 returned error can't find the container with id e8c412eac4ba9286d804aac23630bedc5800b550c0a0ca325975c2790bde4c59
Sep 09 16:00:00.456806 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:00.456780    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-482dce-29816160-2nzdm"]
Sep 09 16:00:00.459740 ip-10-0-129-170 kubenswrapper[2553]: W0909 16:00:00.459698    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8cd335a_399b_4e1f_a515_2628066ccf14.slice/crio-f0a6caa0a544859dfb10cd3704912053472e8f70b813be4dd04c86e9b7638ef2 WatchSource:0}: Error finding container f0a6caa0a544859dfb10cd3704912053472e8f70b813be4dd04c86e9b7638ef2: Status 404 returned error can't find the container with id f0a6caa0a544859dfb10cd3704912053472e8f70b813be4dd04c86e9b7638ef2
Sep 09 16:00:00.493113 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:00.493085    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a2b442-29816160-xs9sc"]
Sep 09 16:00:00.495203 ip-10-0-129-170 kubenswrapper[2553]: W0909 16:00:00.495181    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf79a4296_34a1_4d8f_a818_2b2cfbec9bb9.slice/crio-59775c02c7ff364298768afb50a22e0ba53cad34156334d28328971b9b215ae4 WatchSource:0}: Error finding container 59775c02c7ff364298768afb50a22e0ba53cad34156334d28328971b9b215ae4: Status 404 returned error can't find the container with id 59775c02c7ff364298768afb50a22e0ba53cad34156334d28328971b9b215ae4
Sep 09 16:00:01.427237 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:01.427201    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a2b442-29816160-xs9sc" event={"ID":"f79a4296-34a1-4d8f-a818-2b2cfbec9bb9","Type":"ContainerStarted","Data":"232f18da5cee02af50789dc1b47ac11252ad3d9be4b7ca2bf4f46cdd61bddc59"}
Sep 09 16:00:01.427786 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:01.427266    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a2b442-29816160-xs9sc" event={"ID":"f79a4296-34a1-4d8f-a818-2b2cfbec9bb9","Type":"ContainerStarted","Data":"59775c02c7ff364298768afb50a22e0ba53cad34156334d28328971b9b215ae4"}
Sep 09 16:00:01.428848 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:01.428824    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816160-mpgjw" event={"ID":"25e8a0ab-9725-47d2-9c06-5cb0db65f98e","Type":"ContainerStarted","Data":"951ee43b6864065f81e0a865eccc7115a5f4daa2c759049e420f4dd051ee830e"}
Sep 09 16:00:01.428970 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:01.428869    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816160-mpgjw" event={"ID":"25e8a0ab-9725-47d2-9c06-5cb0db65f98e","Type":"ContainerStarted","Data":"e8c412eac4ba9286d804aac23630bedc5800b550c0a0ca325975c2790bde4c59"}
Sep 09 16:00:01.430269 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:01.430239    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-482dce-29816160-2nzdm" event={"ID":"c8cd335a-399b-4e1f-a515-2628066ccf14","Type":"ContainerStarted","Data":"c6e62d76be344a73eaf75c04123bae588a698b6cf6dbfcab89bc9e07bf77b036"}
Sep 09 16:00:01.430373 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:01.430286    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-482dce-29816160-2nzdm" event={"ID":"c8cd335a-399b-4e1f-a515-2628066ccf14","Type":"ContainerStarted","Data":"f0a6caa0a544859dfb10cd3704912053472e8f70b813be4dd04c86e9b7638ef2"}
Sep 09 16:00:01.443783 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:01.443670    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a2b442-29816160-xs9sc" podStartSLOduration=1.443650042 podStartE2EDuration="1.443650042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:00:01.440975277 +0000 UTC m=+2367.039869923" watchObservedRunningTime="2026-09-09 16:00:01.443650042 +0000 UTC m=+2367.042544686"
Sep 09 16:00:01.455117 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:01.455066    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-482dce-29816160-2nzdm" podStartSLOduration=1.455042329 podStartE2EDuration="1.455042329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:00:01.454593138 +0000 UTC m=+2367.053487784" watchObservedRunningTime="2026-09-09 16:00:01.455042329 +0000 UTC m=+2367.053936973"
Sep 09 16:00:01.469085 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:01.469022    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816160-mpgjw" podStartSLOduration=1.469005289 podStartE2EDuration="1.469005289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:00:01.46765113 +0000 UTC m=+2367.066545773" watchObservedRunningTime="2026-09-09 16:00:01.469005289 +0000 UTC m=+2367.067899933"
Sep 09 16:00:02.436230 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:02.436192    2553 generic.go:356] "Generic (PLEG): container finished" podID="f79a4296-34a1-4d8f-a818-2b2cfbec9bb9" containerID="232f18da5cee02af50789dc1b47ac11252ad3d9be4b7ca2bf4f46cdd61bddc59" exitCode=0
Sep 09 16:00:02.436663 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:02.436281    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a2b442-29816160-xs9sc" event={"ID":"f79a4296-34a1-4d8f-a818-2b2cfbec9bb9","Type":"ContainerDied","Data":"232f18da5cee02af50789dc1b47ac11252ad3d9be4b7ca2bf4f46cdd61bddc59"}
Sep 09 16:00:02.437827 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:02.437798    2553 generic.go:356] "Generic (PLEG): container finished" podID="25e8a0ab-9725-47d2-9c06-5cb0db65f98e" containerID="951ee43b6864065f81e0a865eccc7115a5f4daa2c759049e420f4dd051ee830e" exitCode=0
Sep 09 16:00:02.437973 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:02.437828    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816160-mpgjw" event={"ID":"25e8a0ab-9725-47d2-9c06-5cb0db65f98e","Type":"ContainerDied","Data":"951ee43b6864065f81e0a865eccc7115a5f4daa2c759049e420f4dd051ee830e"}
Sep 09 16:00:02.439486 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:02.439464    2553 generic.go:356] "Generic (PLEG): container finished" podID="c8cd335a-399b-4e1f-a515-2628066ccf14" containerID="c6e62d76be344a73eaf75c04123bae588a698b6cf6dbfcab89bc9e07bf77b036" exitCode=0
Sep 09 16:00:02.439574 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:02.439539    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-482dce-29816160-2nzdm" event={"ID":"c8cd335a-399b-4e1f-a515-2628066ccf14","Type":"ContainerDied","Data":"c6e62d76be344a73eaf75c04123bae588a698b6cf6dbfcab89bc9e07bf77b036"}
Sep 09 16:00:03.326670 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:03.326629    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"]
Sep 09 16:00:03.326949 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:03.326919    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm" podUID="32496718-d42f-49cd-b024-9a98c0811354" containerName="istio-proxy" containerID="cri-o://baa05a173beb9194e3f70071b0dd6d52a84478fbb9a2c9058c2110b6e8d20509" gracePeriod=30
Sep 09 16:00:03.598214 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:03.598181    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-482dce-29816160-2nzdm"
Sep 09 16:00:03.626775 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:03.626744    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a2b442-29816160-xs9sc"
Sep 09 16:00:03.647856 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:03.647815    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816160-mpgjw"
Sep 09 16:00:03.709424 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:03.709386    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a2b442-khdzc"]
Sep 09 16:00:03.709905 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:03.709849    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8cd335a-399b-4e1f-a515-2628066ccf14" containerName="cleanup"
Sep 09 16:00:03.709905 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:03.709870    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8cd335a-399b-4e1f-a515-2628066ccf14" containerName="cleanup"
Sep 09 16:00:03.709905 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:03.709888    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f79a4296-34a1-4d8f-a818-2b2cfbec9bb9" containerName="cleanup"
Sep 09 16:00:03.709905 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:03.709896    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f79a4296-34a1-4d8f-a818-2b2cfbec9bb9" containerName="cleanup"
Sep 09 16:00:03.710179 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:03.709918    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25e8a0ab-9725-47d2-9c06-5cb0db65f98e" containerName="cleanup"
Sep 09 16:00:03.710179 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:03.709927    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="25e8a0ab-9725-47d2-9c06-5cb0db65f98e" containerName="cleanup"
Sep 09 16:00:03.710179 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:03.710040    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f79a4296-34a1-4d8f-a818-2b2cfbec9bb9" containerName="cleanup"
Sep 09 16:00:03.710179 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:03.710053    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="25e8a0ab-9725-47d2-9c06-5cb0db65f98e" containerName="cleanup"
Sep 09 16:00:03.710179 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:03.710060    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8cd335a-399b-4e1f-a515-2628066ccf14" containerName="cleanup"
Sep 09 16:00:03.712957 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:03.712938    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a2b442-khdzc"
Sep 09 16:00:03.715441 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:03.715421    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 16:00:03.720359 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:03.720330    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a2b442-khdzc"]
Sep 09 16:00:03.825178 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:03.825128    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b430a3f-85be-4a31-b80f-6b3fa9774fdf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a2b442-khdzc\" (UID: \"6b430a3f-85be-4a31-b80f-6b3fa9774fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a2b442-khdzc"
Sep 09 16:00:03.926523 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:03.926491    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b430a3f-85be-4a31-b80f-6b3fa9774fdf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a2b442-khdzc\" (UID: \"6b430a3f-85be-4a31-b80f-6b3fa9774fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a2b442-khdzc"
Sep 09 16:00:03.927236 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:03.927210    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b430a3f-85be-4a31-b80f-6b3fa9774fdf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a2b442-khdzc\" (UID: \"6b430a3f-85be-4a31-b80f-6b3fa9774fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a2b442-khdzc"
Sep 09 16:00:04.024563 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:04.024525    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a2b442-khdzc"
Sep 09 16:00:04.297465 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:04.297439    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a2b442-khdzc"]
Sep 09 16:00:04.299969 ip-10-0-129-170 kubenswrapper[2553]: W0909 16:00:04.299942    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b430a3f_85be_4a31_b80f_6b3fa9774fdf.slice/crio-0187c58634d7f0d40b97cd03feb034b2dd8aec5716aae3e3c0415e341334a04b WatchSource:0}: Error finding container 0187c58634d7f0d40b97cd03feb034b2dd8aec5716aae3e3c0415e341334a04b: Status 404 returned error can't find the container with id 0187c58634d7f0d40b97cd03feb034b2dd8aec5716aae3e3c0415e341334a04b
Sep 09 16:00:04.452629 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:04.452600    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a2b442-29816160-xs9sc"
Sep 09 16:00:04.452927 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:04.452606    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a2b442-29816160-xs9sc" event={"ID":"f79a4296-34a1-4d8f-a818-2b2cfbec9bb9","Type":"ContainerDied","Data":"59775c02c7ff364298768afb50a22e0ba53cad34156334d28328971b9b215ae4"}
Sep 09 16:00:04.452927 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:04.452794    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59775c02c7ff364298768afb50a22e0ba53cad34156334d28328971b9b215ae4"
Sep 09 16:00:04.454479 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:04.454458    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816160-mpgjw"
Sep 09 16:00:04.454625 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:04.454483    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816160-mpgjw" event={"ID":"25e8a0ab-9725-47d2-9c06-5cb0db65f98e","Type":"ContainerDied","Data":"e8c412eac4ba9286d804aac23630bedc5800b550c0a0ca325975c2790bde4c59"}
Sep 09 16:00:04.454625 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:04.454528    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8c412eac4ba9286d804aac23630bedc5800b550c0a0ca325975c2790bde4c59"
Sep 09 16:00:04.456745 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:04.456703    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-482dce-29816160-2nzdm" event={"ID":"c8cd335a-399b-4e1f-a515-2628066ccf14","Type":"ContainerDied","Data":"f0a6caa0a544859dfb10cd3704912053472e8f70b813be4dd04c86e9b7638ef2"}
Sep 09 16:00:04.456855 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:04.456755    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-482dce-29816160-2nzdm"
Sep 09 16:00:04.456914 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:04.456772    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0a6caa0a544859dfb10cd3704912053472e8f70b813be4dd04c86e9b7638ef2"
Sep 09 16:00:04.458348 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:04.458319    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a2b442-khdzc" event={"ID":"6b430a3f-85be-4a31-b80f-6b3fa9774fdf","Type":"ContainerStarted","Data":"0187c58634d7f0d40b97cd03feb034b2dd8aec5716aae3e3c0415e341334a04b"}
Sep 09 16:00:05.466119 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:05.466083    2553 generic.go:356] "Generic (PLEG): container finished" podID="6b430a3f-85be-4a31-b80f-6b3fa9774fdf" containerID="89a527d5ababa71ad2214178b7a1c7edf9eaa7d1ff23c45200372efdbb8497f9" exitCode=0
Sep 09 16:00:05.466592 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:05.466145    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a2b442-khdzc" event={"ID":"6b430a3f-85be-4a31-b80f-6b3fa9774fdf","Type":"ContainerDied","Data":"89a527d5ababa71ad2214178b7a1c7edf9eaa7d1ff23c45200372efdbb8497f9"}
Sep 09 16:00:06.607879 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:06.607850    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a2b442-khdzc"
Sep 09 16:00:06.753477 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:06.753382    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b430a3f-85be-4a31-b80f-6b3fa9774fdf-maas-api-service-ca\") pod \"6b430a3f-85be-4a31-b80f-6b3fa9774fdf\" (UID: \"6b430a3f-85be-4a31-b80f-6b3fa9774fdf\") "
Sep 09 16:00:06.753800 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:06.753778    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b430a3f-85be-4a31-b80f-6b3fa9774fdf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6b430a3f-85be-4a31-b80f-6b3fa9774fdf" (UID: "6b430a3f-85be-4a31-b80f-6b3fa9774fdf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:00:06.854556 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:06.854513    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b430a3f-85be-4a31-b80f-6b3fa9774fdf-maas-api-service-ca\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 16:00:07.477928 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:07.477896    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a2b442-khdzc"
Sep 09 16:00:07.477928 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:07.477918    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a2b442-khdzc" event={"ID":"6b430a3f-85be-4a31-b80f-6b3fa9774fdf","Type":"ContainerDied","Data":"0187c58634d7f0d40b97cd03feb034b2dd8aec5716aae3e3c0415e341334a04b"}
Sep 09 16:00:07.478147 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:07.477971    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0187c58634d7f0d40b97cd03feb034b2dd8aec5716aae3e3c0415e341334a04b"
Sep 09 16:00:08.485924 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:08.485893    2553 generic.go:356] "Generic (PLEG): container finished" podID="32496718-d42f-49cd-b024-9a98c0811354" containerID="baa05a173beb9194e3f70071b0dd6d52a84478fbb9a2c9058c2110b6e8d20509" exitCode=0
Sep 09 16:00:08.486331 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:08.485973    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm" event={"ID":"32496718-d42f-49cd-b024-9a98c0811354","Type":"ContainerDied","Data":"baa05a173beb9194e3f70071b0dd6d52a84478fbb9a2c9058c2110b6e8d20509"}
Sep 09 16:00:08.486331 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:08.486044    2553 scope.go:117] "RemoveContainer" containerID="00dad01c3282c9bef05cd83c4e6ff033383083a9befeba9819394c0d55df9535"
Sep 09 16:00:08.589156 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:08.589088    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"
Sep 09 16:00:08.672338 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:08.672301    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32496718-d42f-49cd-b024-9a98c0811354-credential-socket\") pod \"32496718-d42f-49cd-b024-9a98c0811354\" (UID: \"32496718-d42f-49cd-b024-9a98c0811354\") "
Sep 09 16:00:08.672520 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:08.672356    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32496718-d42f-49cd-b024-9a98c0811354-istio-data\") pod \"32496718-d42f-49cd-b024-9a98c0811354\" (UID: \"32496718-d42f-49cd-b024-9a98c0811354\") "
Sep 09 16:00:08.672520 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:08.672375    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32496718-d42f-49cd-b024-9a98c0811354-workload-certs\") pod \"32496718-d42f-49cd-b024-9a98c0811354\" (UID: \"32496718-d42f-49cd-b024-9a98c0811354\") "
Sep 09 16:00:08.672520 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:08.672404    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32496718-d42f-49cd-b024-9a98c0811354-istio-podinfo\") pod \"32496718-d42f-49cd-b024-9a98c0811354\" (UID: \"32496718-d42f-49cd-b024-9a98c0811354\") "
Sep 09 16:00:08.672520 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:08.672419    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32496718-d42f-49cd-b024-9a98c0811354-istio-envoy\") pod \"32496718-d42f-49cd-b024-9a98c0811354\" (UID: \"32496718-d42f-49cd-b024-9a98c0811354\") "
Sep 09 16:00:08.672520 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:08.672457    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32496718-d42f-49cd-b024-9a98c0811354-istiod-ca-cert\") pod \"32496718-d42f-49cd-b024-9a98c0811354\" (UID: \"32496718-d42f-49cd-b024-9a98c0811354\") "
Sep 09 16:00:08.672520 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:08.672513    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9xd8\" (UniqueName: \"kubernetes.io/projected/32496718-d42f-49cd-b024-9a98c0811354-kube-api-access-t9xd8\") pod \"32496718-d42f-49cd-b024-9a98c0811354\" (UID: \"32496718-d42f-49cd-b024-9a98c0811354\") "
Sep 09 16:00:08.672875 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:08.672536    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32496718-d42f-49cd-b024-9a98c0811354-istio-token\") pod \"32496718-d42f-49cd-b024-9a98c0811354\" (UID: \"32496718-d42f-49cd-b024-9a98c0811354\") "
Sep 09 16:00:08.672875 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:08.672615    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32496718-d42f-49cd-b024-9a98c0811354-workload-socket\") pod \"32496718-d42f-49cd-b024-9a98c0811354\" (UID: \"32496718-d42f-49cd-b024-9a98c0811354\") "
Sep 09 16:00:08.672875 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:08.672651    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32496718-d42f-49cd-b024-9a98c0811354-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "32496718-d42f-49cd-b024-9a98c0811354" (UID: "32496718-d42f-49cd-b024-9a98c0811354"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:00:08.672875 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:08.672666    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32496718-d42f-49cd-b024-9a98c0811354-istio-data" (OuterVolumeSpecName: "istio-data") pod "32496718-d42f-49cd-b024-9a98c0811354" (UID: "32496718-d42f-49cd-b024-9a98c0811354"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:00:08.672875 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:08.672758    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32496718-d42f-49cd-b024-9a98c0811354-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "32496718-d42f-49cd-b024-9a98c0811354" (UID: "32496718-d42f-49cd-b024-9a98c0811354"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:00:08.673145 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:08.672932    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32496718-d42f-49cd-b024-9a98c0811354-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "32496718-d42f-49cd-b024-9a98c0811354" (UID: "32496718-d42f-49cd-b024-9a98c0811354"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:00:08.673145 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:08.672938    2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32496718-d42f-49cd-b024-9a98c0811354-credential-socket\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 16:00:08.673145 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:08.672964    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32496718-d42f-49cd-b024-9a98c0811354-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "32496718-d42f-49cd-b024-9a98c0811354" (UID: "32496718-d42f-49cd-b024-9a98c0811354"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:00:08.673145 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:08.672984    2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32496718-d42f-49cd-b024-9a98c0811354-istio-data\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 16:00:08.673145 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:08.673001    2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32496718-d42f-49cd-b024-9a98c0811354-workload-certs\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 16:00:08.675006 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:08.674980    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32496718-d42f-49cd-b024-9a98c0811354-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "32496718-d42f-49cd-b024-9a98c0811354" (UID: "32496718-d42f-49cd-b024-9a98c0811354"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 16:00:08.675006 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:08.674986    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32496718-d42f-49cd-b024-9a98c0811354-kube-api-access-t9xd8" (OuterVolumeSpecName: "kube-api-access-t9xd8") pod "32496718-d42f-49cd-b024-9a98c0811354" (UID: "32496718-d42f-49cd-b024-9a98c0811354"). InnerVolumeSpecName "kube-api-access-t9xd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:00:08.675006 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:08.675000    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32496718-d42f-49cd-b024-9a98c0811354-istio-token" (OuterVolumeSpecName: "istio-token") pod "32496718-d42f-49cd-b024-9a98c0811354" (UID: "32496718-d42f-49cd-b024-9a98c0811354"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:00:08.675198 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:08.675032    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32496718-d42f-49cd-b024-9a98c0811354-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "32496718-d42f-49cd-b024-9a98c0811354" (UID: "32496718-d42f-49cd-b024-9a98c0811354"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:00:08.774425 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:08.774384    2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32496718-d42f-49cd-b024-9a98c0811354-workload-socket\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 16:00:08.774425 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:08.774416    2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32496718-d42f-49cd-b024-9a98c0811354-istio-podinfo\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 16:00:08.774425 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:08.774425    2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32496718-d42f-49cd-b024-9a98c0811354-istio-envoy\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 16:00:08.774425 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:08.774433    2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32496718-d42f-49cd-b024-9a98c0811354-istiod-ca-cert\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 16:00:08.774705 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:08.774444    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9xd8\" (UniqueName: \"kubernetes.io/projected/32496718-d42f-49cd-b024-9a98c0811354-kube-api-access-t9xd8\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 16:00:08.774705 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:08.774453    2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32496718-d42f-49cd-b024-9a98c0811354-istio-token\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 16:00:09.491985 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:09.491956    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"
Sep 09 16:00:09.492501 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:09.491956    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm" event={"ID":"32496718-d42f-49cd-b024-9a98c0811354","Type":"ContainerDied","Data":"499890a8595a741c901c1a8e9d7c093cc4cdb30da82d8b23a831ffc178a096ee"}
Sep 09 16:00:09.492501 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:09.492093    2553 scope.go:117] "RemoveContainer" containerID="baa05a173beb9194e3f70071b0dd6d52a84478fbb9a2c9058c2110b6e8d20509"
Sep 09 16:00:09.509047 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:09.509018    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"]
Sep 09 16:00:09.513048 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:09.513021    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-1218fe-openshift-default-f7c4db968-5jwtm"]
Sep 09 16:00:10.962143 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:10.962109    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32496718-d42f-49cd-b024-9a98c0811354" path="/var/lib/kubelet/pods/32496718-d42f-49cd-b024-9a98c0811354/volumes"
Sep 09 16:00:14.391004 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:14.390943    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a2b442-29816160-xs9sc"]
Sep 09 16:00:14.395589 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:14.395558    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a2b442-29816160-xs9sc"]
Sep 09 16:00:14.414974 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:14.414941    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a2b442-9cbbd6765-4qnxr"]
Sep 09 16:00:14.415234 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:14.415182    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2b442-9cbbd6765-4qnxr" podUID="0452b747-ebc3-4807-87c3-927b2be91b08" containerName="payload-processing" containerID="cri-o://44d801ecaf950125f68f3284f5950c6c44b45556c00a40a6d76e56a6dea31285" gracePeriod=30
Sep 09 16:00:14.423823 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:14.423799    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a2b442-7887cc56f8-xkk4q"]
Sep 09 16:00:14.424051 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:14.424009    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2b442-7887cc56f8-xkk4q" podUID="d74849df-dcca-410f-93f2-7b84f72e6b59" containerName="payload-pre-processing" containerID="cri-o://636ee8cb01d014ce7625066ae3f2b64e530ceb692b7014a4fca0e37ce6c65f4c" gracePeriod=30
Sep 09 16:00:14.627459 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:14.627429    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a2b442-khdzc"]
Sep 09 16:00:14.634549 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:14.634429    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a2b442-khdzc"]
Sep 09 16:00:14.695644 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:14.695621    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2b442-9cbbd6765-4qnxr"
Sep 09 16:00:14.698095 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:14.698070    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2b442-7887cc56f8-xkk4q"
Sep 09 16:00:14.828757 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:14.828686    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xndnj\" (UniqueName: \"kubernetes.io/projected/d74849df-dcca-410f-93f2-7b84f72e6b59-kube-api-access-xndnj\") pod \"d74849df-dcca-410f-93f2-7b84f72e6b59\" (UID: \"d74849df-dcca-410f-93f2-7b84f72e6b59\") "
Sep 09 16:00:14.828965 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:14.828796    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0452b747-ebc3-4807-87c3-927b2be91b08-plugins-config-volume\") pod \"0452b747-ebc3-4807-87c3-927b2be91b08\" (UID: \"0452b747-ebc3-4807-87c3-927b2be91b08\") "
Sep 09 16:00:14.828965 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:14.828865    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d74849df-dcca-410f-93f2-7b84f72e6b59-plugins-config-volume\") pod \"d74849df-dcca-410f-93f2-7b84f72e6b59\" (UID: \"d74849df-dcca-410f-93f2-7b84f72e6b59\") "
Sep 09 16:00:14.828965 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:14.828901    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shqmj\" (UniqueName: \"kubernetes.io/projected/0452b747-ebc3-4807-87c3-927b2be91b08-kube-api-access-shqmj\") pod \"0452b747-ebc3-4807-87c3-927b2be91b08\" (UID: \"0452b747-ebc3-4807-87c3-927b2be91b08\") "
Sep 09 16:00:14.829174 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:14.829150    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0452b747-ebc3-4807-87c3-927b2be91b08-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0452b747-ebc3-4807-87c3-927b2be91b08" (UID: "0452b747-ebc3-4807-87c3-927b2be91b08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:00:14.829314 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:14.829290    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d74849df-dcca-410f-93f2-7b84f72e6b59-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d74849df-dcca-410f-93f2-7b84f72e6b59" (UID: "d74849df-dcca-410f-93f2-7b84f72e6b59"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:00:14.830851 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:14.830828    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d74849df-dcca-410f-93f2-7b84f72e6b59-kube-api-access-xndnj" (OuterVolumeSpecName: "kube-api-access-xndnj") pod "d74849df-dcca-410f-93f2-7b84f72e6b59" (UID: "d74849df-dcca-410f-93f2-7b84f72e6b59"). InnerVolumeSpecName "kube-api-access-xndnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:00:14.830975 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:14.830958    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0452b747-ebc3-4807-87c3-927b2be91b08-kube-api-access-shqmj" (OuterVolumeSpecName: "kube-api-access-shqmj") pod "0452b747-ebc3-4807-87c3-927b2be91b08" (UID: "0452b747-ebc3-4807-87c3-927b2be91b08"). InnerVolumeSpecName "kube-api-access-shqmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:00:14.930631 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:14.930538    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xndnj\" (UniqueName: \"kubernetes.io/projected/d74849df-dcca-410f-93f2-7b84f72e6b59-kube-api-access-xndnj\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 16:00:14.930631 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:14.930575    2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0452b747-ebc3-4807-87c3-927b2be91b08-plugins-config-volume\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 16:00:14.930631 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:14.930586    2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d74849df-dcca-410f-93f2-7b84f72e6b59-plugins-config-volume\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 16:00:14.930631 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:14.930595    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shqmj\" (UniqueName: \"kubernetes.io/projected/0452b747-ebc3-4807-87c3-927b2be91b08-kube-api-access-shqmj\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 16:00:14.962449 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:14.962413    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b430a3f-85be-4a31-b80f-6b3fa9774fdf" path="/var/lib/kubelet/pods/6b430a3f-85be-4a31-b80f-6b3fa9774fdf/volumes"
Sep 09 16:00:14.962710 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:14.962697    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f79a4296-34a1-4d8f-a818-2b2cfbec9bb9" path="/var/lib/kubelet/pods/f79a4296-34a1-4d8f-a818-2b2cfbec9bb9/volumes"
Sep 09 16:00:15.531376 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:15.531341    2553 generic.go:356] "Generic (PLEG): container finished" podID="d74849df-dcca-410f-93f2-7b84f72e6b59" containerID="636ee8cb01d014ce7625066ae3f2b64e530ceb692b7014a4fca0e37ce6c65f4c" exitCode=0
Sep 09 16:00:15.531855 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:15.531387    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2b442-7887cc56f8-xkk4q"
Sep 09 16:00:15.531855 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:15.531425    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2b442-7887cc56f8-xkk4q" event={"ID":"d74849df-dcca-410f-93f2-7b84f72e6b59","Type":"ContainerDied","Data":"636ee8cb01d014ce7625066ae3f2b64e530ceb692b7014a4fca0e37ce6c65f4c"}
Sep 09 16:00:15.531855 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:15.531477    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2b442-7887cc56f8-xkk4q" event={"ID":"d74849df-dcca-410f-93f2-7b84f72e6b59","Type":"ContainerDied","Data":"339b9ab0b20ba811301c287144073dbd7e92cb4e584ce603fbe58ccdb318496b"}
Sep 09 16:00:15.531855 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:15.531498    2553 scope.go:117] "RemoveContainer" containerID="636ee8cb01d014ce7625066ae3f2b64e530ceb692b7014a4fca0e37ce6c65f4c"
Sep 09 16:00:15.532987 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:15.532964    2553 generic.go:356] "Generic (PLEG): container finished" podID="0452b747-ebc3-4807-87c3-927b2be91b08" containerID="44d801ecaf950125f68f3284f5950c6c44b45556c00a40a6d76e56a6dea31285" exitCode=0
Sep 09 16:00:15.533106 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:15.533007    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2b442-9cbbd6765-4qnxr" event={"ID":"0452b747-ebc3-4807-87c3-927b2be91b08","Type":"ContainerDied","Data":"44d801ecaf950125f68f3284f5950c6c44b45556c00a40a6d76e56a6dea31285"}
Sep 09 16:00:15.533106 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:15.533016    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2b442-9cbbd6765-4qnxr"
Sep 09 16:00:15.533106 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:15.533043    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2b442-9cbbd6765-4qnxr" event={"ID":"0452b747-ebc3-4807-87c3-927b2be91b08","Type":"ContainerDied","Data":"4c41503053ed0defdaa712bca3187484f6c1242761a224d8f7f292729c84b04c"}
Sep 09 16:00:15.540470 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:15.540452    2553 scope.go:117] "RemoveContainer" containerID="636ee8cb01d014ce7625066ae3f2b64e530ceb692b7014a4fca0e37ce6c65f4c"
Sep 09 16:00:15.540815 ip-10-0-129-170 kubenswrapper[2553]: E0909 16:00:15.540785    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"636ee8cb01d014ce7625066ae3f2b64e530ceb692b7014a4fca0e37ce6c65f4c\": container with ID starting with 636ee8cb01d014ce7625066ae3f2b64e530ceb692b7014a4fca0e37ce6c65f4c not found: ID does not exist" containerID="636ee8cb01d014ce7625066ae3f2b64e530ceb692b7014a4fca0e37ce6c65f4c"
Sep 09 16:00:15.540900 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:15.540827    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"636ee8cb01d014ce7625066ae3f2b64e530ceb692b7014a4fca0e37ce6c65f4c"} err="failed to get container status \"636ee8cb01d014ce7625066ae3f2b64e530ceb692b7014a4fca0e37ce6c65f4c\": rpc error: code = NotFound desc = could not find container \"636ee8cb01d014ce7625066ae3f2b64e530ceb692b7014a4fca0e37ce6c65f4c\": container with ID starting with 636ee8cb01d014ce7625066ae3f2b64e530ceb692b7014a4fca0e37ce6c65f4c not found: ID does not exist"
Sep 09 16:00:15.540900 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:15.540849    2553 scope.go:117] "RemoveContainer" containerID="44d801ecaf950125f68f3284f5950c6c44b45556c00a40a6d76e56a6dea31285"
Sep 09 16:00:15.548568 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:15.548543    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a2b442-7887cc56f8-xkk4q"]
Sep 09 16:00:15.549821 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:15.549803    2553 scope.go:117] "RemoveContainer" containerID="44d801ecaf950125f68f3284f5950c6c44b45556c00a40a6d76e56a6dea31285"
Sep 09 16:00:15.550188 ip-10-0-129-170 kubenswrapper[2553]: E0909 16:00:15.550166    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44d801ecaf950125f68f3284f5950c6c44b45556c00a40a6d76e56a6dea31285\": container with ID starting with 44d801ecaf950125f68f3284f5950c6c44b45556c00a40a6d76e56a6dea31285 not found: ID does not exist" containerID="44d801ecaf950125f68f3284f5950c6c44b45556c00a40a6d76e56a6dea31285"
Sep 09 16:00:15.550257 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:15.550202    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44d801ecaf950125f68f3284f5950c6c44b45556c00a40a6d76e56a6dea31285"} err="failed to get container status \"44d801ecaf950125f68f3284f5950c6c44b45556c00a40a6d76e56a6dea31285\": rpc error: code = NotFound desc = could not find container \"44d801ecaf950125f68f3284f5950c6c44b45556c00a40a6d76e56a6dea31285\": container with ID starting with 44d801ecaf950125f68f3284f5950c6c44b45556c00a40a6d76e56a6dea31285 not found: ID does not exist"
Sep 09 16:00:15.550800 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:15.550777    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a2b442-7887cc56f8-xkk4q"]
Sep 09 16:00:15.560338 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:15.560314    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a2b442-9cbbd6765-4qnxr"]
Sep 09 16:00:15.564088 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:15.564067    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a2b442-9cbbd6765-4qnxr"]
Sep 09 16:00:16.967030 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:16.962818    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0452b747-ebc3-4807-87c3-927b2be91b08" path="/var/lib/kubelet/pods/0452b747-ebc3-4807-87c3-927b2be91b08/volumes"
Sep 09 16:00:16.967030 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:16.963440    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d74849df-dcca-410f-93f2-7b84f72e6b59" path="/var/lib/kubelet/pods/d74849df-dcca-410f-93f2-7b84f72e6b59/volumes"
Sep 09 16:00:21.546260 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:21.546214    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-450a51-64lkb"]
Sep 09 16:00:21.546774 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:21.546695    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d74849df-dcca-410f-93f2-7b84f72e6b59" containerName="payload-pre-processing"
Sep 09 16:00:21.546774 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:21.546714    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="d74849df-dcca-410f-93f2-7b84f72e6b59" containerName="payload-pre-processing"
Sep 09 16:00:21.546774 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:21.546755    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32496718-d42f-49cd-b024-9a98c0811354" containerName="istio-proxy"
Sep 09 16:00:21.546774 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:21.546764    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="32496718-d42f-49cd-b024-9a98c0811354" containerName="istio-proxy"
Sep 09 16:00:21.546998 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:21.546789    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b430a3f-85be-4a31-b80f-6b3fa9774fdf" containerName="revoke-keys"
Sep 09 16:00:21.546998 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:21.546798    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b430a3f-85be-4a31-b80f-6b3fa9774fdf" containerName="revoke-keys"
Sep 09 16:00:21.546998 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:21.546817    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0452b747-ebc3-4807-87c3-927b2be91b08" containerName="payload-processing"
Sep 09 16:00:21.546998 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:21.546826    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0452b747-ebc3-4807-87c3-927b2be91b08" containerName="payload-processing"
Sep 09 16:00:21.546998 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:21.546910    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="32496718-d42f-49cd-b024-9a98c0811354" containerName="istio-proxy"
Sep 09 16:00:21.546998 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:21.546923    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="0452b747-ebc3-4807-87c3-927b2be91b08" containerName="payload-processing"
Sep 09 16:00:21.546998 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:21.546933    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="32496718-d42f-49cd-b024-9a98c0811354" containerName="istio-proxy"
Sep 09 16:00:21.546998 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:21.546943    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="d74849df-dcca-410f-93f2-7b84f72e6b59" containerName="payload-pre-processing"
Sep 09 16:00:21.546998 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:21.546952    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b430a3f-85be-4a31-b80f-6b3fa9774fdf" containerName="revoke-keys"
Sep 09 16:00:21.551517 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:21.551497    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-450a51-64lkb"
Sep 09 16:00:21.554575 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:21.554551    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2zlsf\""
Sep 09 16:00:21.555179 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:21.555153    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 16:00:21.555986 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:21.555960    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-450a51-64lkb"]
Sep 09 16:00:21.694692 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:21.694648    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43495075-fd37-498f-8fe6-1fe6ff055b7c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-450a51-64lkb\" (UID: \"43495075-fd37-498f-8fe6-1fe6ff055b7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-450a51-64lkb"
Sep 09 16:00:21.795399 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:21.795359    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43495075-fd37-498f-8fe6-1fe6ff055b7c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-450a51-64lkb\" (UID: \"43495075-fd37-498f-8fe6-1fe6ff055b7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-450a51-64lkb"
Sep 09 16:00:21.796058 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:21.796038    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43495075-fd37-498f-8fe6-1fe6ff055b7c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-450a51-64lkb\" (UID: \"43495075-fd37-498f-8fe6-1fe6ff055b7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-450a51-64lkb"
Sep 09 16:00:21.865537 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:21.865426    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-450a51-64lkb"
Sep 09 16:00:22.138054 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:22.137959    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-450a51-64lkb"]
Sep 09 16:00:22.141941 ip-10-0-129-170 kubenswrapper[2553]: W0909 16:00:22.141908    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43495075_fd37_498f_8fe6_1fe6ff055b7c.slice/crio-001c20bc8c7e11489cba0c0027d93e416a99d95f7896e46fc3ec8e7806eb1cec WatchSource:0}: Error finding container 001c20bc8c7e11489cba0c0027d93e416a99d95f7896e46fc3ec8e7806eb1cec: Status 404 returned error can't find the container with id 001c20bc8c7e11489cba0c0027d93e416a99d95f7896e46fc3ec8e7806eb1cec
Sep 09 16:00:22.573476 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:22.573441    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-450a51-64lkb" event={"ID":"43495075-fd37-498f-8fe6-1fe6ff055b7c","Type":"ContainerStarted","Data":"9ae20dc3416eb852ad1182b7a34500fe123afd9e4eae12dd95499f4dc42e6881"}
Sep 09 16:00:22.574043 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:22.573496    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-450a51-64lkb" event={"ID":"43495075-fd37-498f-8fe6-1fe6ff055b7c","Type":"ContainerStarted","Data":"001c20bc8c7e11489cba0c0027d93e416a99d95f7896e46fc3ec8e7806eb1cec"}
Sep 09 16:00:22.590264 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:22.590213    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-450a51-64lkb" podStartSLOduration=1.590194739 podStartE2EDuration="1.590194739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:00:22.588536304 +0000 UTC m=+2388.187430947" watchObservedRunningTime="2026-09-09 16:00:22.590194739 +0000 UTC m=+2388.189089383"
Sep 09 16:00:23.579311 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:23.579272    2553 generic.go:356] "Generic (PLEG): container finished" podID="43495075-fd37-498f-8fe6-1fe6ff055b7c" containerID="9ae20dc3416eb852ad1182b7a34500fe123afd9e4eae12dd95499f4dc42e6881" exitCode=0
Sep 09 16:00:23.579708 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:23.579364    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-450a51-64lkb" event={"ID":"43495075-fd37-498f-8fe6-1fe6ff055b7c","Type":"ContainerDied","Data":"9ae20dc3416eb852ad1182b7a34500fe123afd9e4eae12dd95499f4dc42e6881"}
Sep 09 16:00:24.713862 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:24.713833    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-450a51-64lkb"
Sep 09 16:00:24.718390 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:24.718371    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43495075-fd37-498f-8fe6-1fe6ff055b7c-maas-api-service-ca\") pod \"43495075-fd37-498f-8fe6-1fe6ff055b7c\" (UID: \"43495075-fd37-498f-8fe6-1fe6ff055b7c\") "
Sep 09 16:00:24.718689 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:24.718669    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43495075-fd37-498f-8fe6-1fe6ff055b7c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "43495075-fd37-498f-8fe6-1fe6ff055b7c" (UID: "43495075-fd37-498f-8fe6-1fe6ff055b7c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:00:24.819647 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:24.819605    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43495075-fd37-498f-8fe6-1fe6ff055b7c-maas-api-service-ca\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 16:00:25.590229 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:25.590198    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-450a51-64lkb"
Sep 09 16:00:25.590229 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:25.590208    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-450a51-64lkb" event={"ID":"43495075-fd37-498f-8fe6-1fe6ff055b7c","Type":"ContainerDied","Data":"001c20bc8c7e11489cba0c0027d93e416a99d95f7896e46fc3ec8e7806eb1cec"}
Sep 09 16:00:25.590419 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:25.590259    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="001c20bc8c7e11489cba0c0027d93e416a99d95f7896e46fc3ec8e7806eb1cec"
Sep 09 16:00:32.200547 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:32.200508    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-450a51-64lkb"]
Sep 09 16:00:32.205333 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:32.205302    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-450a51-64lkb"]
Sep 09 16:00:32.962876 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:32.962842    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43495075-fd37-498f-8fe6-1fe6ff055b7c" path="/var/lib/kubelet/pods/43495075-fd37-498f-8fe6-1fe6ff055b7c/volumes"
Sep 09 16:00:35.512173 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:35.512139    2553 scope.go:117] "RemoveContainer" containerID="103fc768c2e9dd9d5067b2da6bef1b6b789c5856d53d2aff1a03c334ef336011"
Sep 09 16:00:39.074527 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:39.074486    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-482dce-r9f4d"]
Sep 09 16:00:39.075021 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:39.074828    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32496718-d42f-49cd-b024-9a98c0811354" containerName="istio-proxy"
Sep 09 16:00:39.075021 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:39.074840    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="32496718-d42f-49cd-b024-9a98c0811354" containerName="istio-proxy"
Sep 09 16:00:39.075021 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:39.074853    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43495075-fd37-498f-8fe6-1fe6ff055b7c" containerName="revoke-keys"
Sep 09 16:00:39.075021 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:39.074859    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="43495075-fd37-498f-8fe6-1fe6ff055b7c" containerName="revoke-keys"
Sep 09 16:00:39.075021 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:39.074909    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="43495075-fd37-498f-8fe6-1fe6ff055b7c" containerName="revoke-keys"
Sep 09 16:00:39.079352 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:39.079335    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-482dce-r9f4d"
Sep 09 16:00:39.083195 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:39.082970    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2zlsf\""
Sep 09 16:00:39.083195 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:39.083076    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 16:00:39.084557 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:39.084527    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-482dce-r9f4d"]
Sep 09 16:00:39.127260 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:39.127226    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae3c812e-ebdf-4fc8-81ef-360154273d84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-482dce-r9f4d\" (UID: \"ae3c812e-ebdf-4fc8-81ef-360154273d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-482dce-r9f4d"
Sep 09 16:00:39.228031 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:39.227993    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae3c812e-ebdf-4fc8-81ef-360154273d84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-482dce-r9f4d\" (UID: \"ae3c812e-ebdf-4fc8-81ef-360154273d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-482dce-r9f4d"
Sep 09 16:00:39.228634 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:39.228610    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae3c812e-ebdf-4fc8-81ef-360154273d84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-482dce-r9f4d\" (UID: \"ae3c812e-ebdf-4fc8-81ef-360154273d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-482dce-r9f4d"
Sep 09 16:00:39.391770 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:39.391664    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-482dce-r9f4d"
Sep 09 16:00:39.663356 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:39.663324    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-482dce-r9f4d"]
Sep 09 16:00:39.665780 ip-10-0-129-170 kubenswrapper[2553]: W0909 16:00:39.665719    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae3c812e_ebdf_4fc8_81ef_360154273d84.slice/crio-e6401defd1997951994ff481b5d465087c67ee6e100795981447191ff21dee65 WatchSource:0}: Error finding container e6401defd1997951994ff481b5d465087c67ee6e100795981447191ff21dee65: Status 404 returned error can't find the container with id e6401defd1997951994ff481b5d465087c67ee6e100795981447191ff21dee65
Sep 09 16:00:40.670666 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:40.670633    2553 generic.go:356] "Generic (PLEG): container finished" podID="ae3c812e-ebdf-4fc8-81ef-360154273d84" containerID="a9f4d7fbe86006ea19584bbbd0585aef370d37432f53a2c89bb35c2fe07d0253" exitCode=0
Sep 09 16:00:40.671050 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:40.670715    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-482dce-r9f4d" event={"ID":"ae3c812e-ebdf-4fc8-81ef-360154273d84","Type":"ContainerDied","Data":"a9f4d7fbe86006ea19584bbbd0585aef370d37432f53a2c89bb35c2fe07d0253"}
Sep 09 16:00:40.671050 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:40.670782    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-482dce-r9f4d" event={"ID":"ae3c812e-ebdf-4fc8-81ef-360154273d84","Type":"ContainerStarted","Data":"e6401defd1997951994ff481b5d465087c67ee6e100795981447191ff21dee65"}
Sep 09 16:00:41.807639 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:41.807610    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-482dce-r9f4d"
Sep 09 16:00:41.852530 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:41.852499    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae3c812e-ebdf-4fc8-81ef-360154273d84-maas-api-service-ca\") pod \"ae3c812e-ebdf-4fc8-81ef-360154273d84\" (UID: \"ae3c812e-ebdf-4fc8-81ef-360154273d84\") "
Sep 09 16:00:41.852884 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:41.852859    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae3c812e-ebdf-4fc8-81ef-360154273d84-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ae3c812e-ebdf-4fc8-81ef-360154273d84" (UID: "ae3c812e-ebdf-4fc8-81ef-360154273d84"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:00:41.954069 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:41.953981    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae3c812e-ebdf-4fc8-81ef-360154273d84-maas-api-service-ca\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 16:00:42.681094 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:42.681058    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-482dce-r9f4d"
Sep 09 16:00:42.681094 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:42.681079    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-482dce-r9f4d" event={"ID":"ae3c812e-ebdf-4fc8-81ef-360154273d84","Type":"ContainerDied","Data":"e6401defd1997951994ff481b5d465087c67ee6e100795981447191ff21dee65"}
Sep 09 16:00:42.681304 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:42.681121    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6401defd1997951994ff481b5d465087c67ee6e100795981447191ff21dee65"
Sep 09 16:00:49.633208 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:49.633176    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-482dce-29816160-2nzdm"]
Sep 09 16:00:49.636357 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:49.636330    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-482dce-29816160-2nzdm"]
Sep 09 16:00:49.663684 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:49.663641    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-482dce-5c5c4c564f-vl84m"]
Sep 09 16:00:49.663953 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:49.663902    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-482dce-5c5c4c564f-vl84m" podUID="cdbe03f8-06de-4b6e-8f5b-960b4ff8a797" containerName="payload-processing" containerID="cri-o://e475807116c278cf76a6a9a8ce18bf9dcac52e6fdb6dc7ddb9c2ec35065e54e3" gracePeriod=30
Sep 09 16:00:49.679927 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:49.679895    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-482dce-5bf7f8dc94-n77km"]
Sep 09 16:00:49.680430 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:49.680155    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-482dce-5bf7f8dc94-n77km" podUID="a5673d1e-abdd-494c-a43b-87e0c0d96f9b" containerName="payload-pre-processing" containerID="cri-o://6ad40de535c01a8b61fadf1a3b4e8de1fea8245fe1df54d0d2a215a3f8c91694" gracePeriod=30
Sep 09 16:00:49.824608 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:49.824522    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-482dce-r9f4d"]
Sep 09 16:00:49.826420 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:49.826397    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-482dce-r9f4d"]
Sep 09 16:00:49.927169 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:49.927139    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-482dce-5c5c4c564f-vl84m"
Sep 09 16:00:49.957003 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:49.956979    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-482dce-5bf7f8dc94-n77km"
Sep 09 16:00:50.027429 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:50.027401    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wc468\" (UniqueName: \"kubernetes.io/projected/cdbe03f8-06de-4b6e-8f5b-960b4ff8a797-kube-api-access-wc468\") pod \"cdbe03f8-06de-4b6e-8f5b-960b4ff8a797\" (UID: \"cdbe03f8-06de-4b6e-8f5b-960b4ff8a797\") "
Sep 09 16:00:50.027622 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:50.027498    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5673d1e-abdd-494c-a43b-87e0c0d96f9b-plugins-config-volume\") pod \"a5673d1e-abdd-494c-a43b-87e0c0d96f9b\" (UID: \"a5673d1e-abdd-494c-a43b-87e0c0d96f9b\") "
Sep 09 16:00:50.027622 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:50.027520    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxshp\" (UniqueName: \"kubernetes.io/projected/a5673d1e-abdd-494c-a43b-87e0c0d96f9b-kube-api-access-nxshp\") pod \"a5673d1e-abdd-494c-a43b-87e0c0d96f9b\" (UID: \"a5673d1e-abdd-494c-a43b-87e0c0d96f9b\") "
Sep 09 16:00:50.027622 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:50.027550    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdbe03f8-06de-4b6e-8f5b-960b4ff8a797-plugins-config-volume\") pod \"cdbe03f8-06de-4b6e-8f5b-960b4ff8a797\" (UID: \"cdbe03f8-06de-4b6e-8f5b-960b4ff8a797\") "
Sep 09 16:00:50.027999 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:50.027966    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5673d1e-abdd-494c-a43b-87e0c0d96f9b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a5673d1e-abdd-494c-a43b-87e0c0d96f9b" (UID: "a5673d1e-abdd-494c-a43b-87e0c0d96f9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:00:50.028115 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:50.028020    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdbe03f8-06de-4b6e-8f5b-960b4ff8a797-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cdbe03f8-06de-4b6e-8f5b-960b4ff8a797" (UID: "cdbe03f8-06de-4b6e-8f5b-960b4ff8a797"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:00:50.029777 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:50.029751    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5673d1e-abdd-494c-a43b-87e0c0d96f9b-kube-api-access-nxshp" (OuterVolumeSpecName: "kube-api-access-nxshp") pod "a5673d1e-abdd-494c-a43b-87e0c0d96f9b" (UID: "a5673d1e-abdd-494c-a43b-87e0c0d96f9b"). InnerVolumeSpecName "kube-api-access-nxshp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:00:50.029885 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:50.029793    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdbe03f8-06de-4b6e-8f5b-960b4ff8a797-kube-api-access-wc468" (OuterVolumeSpecName: "kube-api-access-wc468") pod "cdbe03f8-06de-4b6e-8f5b-960b4ff8a797" (UID: "cdbe03f8-06de-4b6e-8f5b-960b4ff8a797"). InnerVolumeSpecName "kube-api-access-wc468". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:00:50.129131 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:50.129088    2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5673d1e-abdd-494c-a43b-87e0c0d96f9b-plugins-config-volume\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 16:00:50.129131 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:50.129122    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxshp\" (UniqueName: \"kubernetes.io/projected/a5673d1e-abdd-494c-a43b-87e0c0d96f9b-kube-api-access-nxshp\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 16:00:50.129131 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:50.129137    2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdbe03f8-06de-4b6e-8f5b-960b4ff8a797-plugins-config-volume\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 16:00:50.129380 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:50.129149    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc468\" (UniqueName: \"kubernetes.io/projected/cdbe03f8-06de-4b6e-8f5b-960b4ff8a797-kube-api-access-wc468\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 16:00:50.726554 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:50.726513    2553 generic.go:356] "Generic (PLEG): container finished" podID="a5673d1e-abdd-494c-a43b-87e0c0d96f9b" containerID="6ad40de535c01a8b61fadf1a3b4e8de1fea8245fe1df54d0d2a215a3f8c91694" exitCode=0
Sep 09 16:00:50.727032 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:50.726568    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-482dce-5bf7f8dc94-n77km"
Sep 09 16:00:50.727032 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:50.726603    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-482dce-5bf7f8dc94-n77km" event={"ID":"a5673d1e-abdd-494c-a43b-87e0c0d96f9b","Type":"ContainerDied","Data":"6ad40de535c01a8b61fadf1a3b4e8de1fea8245fe1df54d0d2a215a3f8c91694"}
Sep 09 16:00:50.727032 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:50.726675    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-482dce-5bf7f8dc94-n77km" event={"ID":"a5673d1e-abdd-494c-a43b-87e0c0d96f9b","Type":"ContainerDied","Data":"5043f50af32ab8861cef5babbf0ba25f7a683e946f86960df9a8d51b0e53ad43"}
Sep 09 16:00:50.727811 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:50.726712    2553 scope.go:117] "RemoveContainer" containerID="6ad40de535c01a8b61fadf1a3b4e8de1fea8245fe1df54d0d2a215a3f8c91694"
Sep 09 16:00:50.728210 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:50.728182    2553 generic.go:356] "Generic (PLEG): container finished" podID="cdbe03f8-06de-4b6e-8f5b-960b4ff8a797" containerID="e475807116c278cf76a6a9a8ce18bf9dcac52e6fdb6dc7ddb9c2ec35065e54e3" exitCode=0
Sep 09 16:00:50.728326 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:50.728225    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-482dce-5c5c4c564f-vl84m"
Sep 09 16:00:50.728326 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:50.728237    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-482dce-5c5c4c564f-vl84m" event={"ID":"cdbe03f8-06de-4b6e-8f5b-960b4ff8a797","Type":"ContainerDied","Data":"e475807116c278cf76a6a9a8ce18bf9dcac52e6fdb6dc7ddb9c2ec35065e54e3"}
Sep 09 16:00:50.728326 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:50.728285    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-482dce-5c5c4c564f-vl84m" event={"ID":"cdbe03f8-06de-4b6e-8f5b-960b4ff8a797","Type":"ContainerDied","Data":"47aeb66b5fc947d1e2a8c18a4eff5078595edadc0a34d84624e5850e95629685"}
Sep 09 16:00:50.737816 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:50.737795    2553 scope.go:117] "RemoveContainer" containerID="6ad40de535c01a8b61fadf1a3b4e8de1fea8245fe1df54d0d2a215a3f8c91694"
Sep 09 16:00:50.738207 ip-10-0-129-170 kubenswrapper[2553]: E0909 16:00:50.738176    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ad40de535c01a8b61fadf1a3b4e8de1fea8245fe1df54d0d2a215a3f8c91694\": container with ID starting with 6ad40de535c01a8b61fadf1a3b4e8de1fea8245fe1df54d0d2a215a3f8c91694 not found: ID does not exist" containerID="6ad40de535c01a8b61fadf1a3b4e8de1fea8245fe1df54d0d2a215a3f8c91694"
Sep 09 16:00:50.738291 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:50.738218    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ad40de535c01a8b61fadf1a3b4e8de1fea8245fe1df54d0d2a215a3f8c91694"} err="failed to get container status \"6ad40de535c01a8b61fadf1a3b4e8de1fea8245fe1df54d0d2a215a3f8c91694\": rpc error: code = NotFound desc = could not find container \"6ad40de535c01a8b61fadf1a3b4e8de1fea8245fe1df54d0d2a215a3f8c91694\": container with ID starting with 6ad40de535c01a8b61fadf1a3b4e8de1fea8245fe1df54d0d2a215a3f8c91694 not found: ID does not exist"
Sep 09 16:00:50.738291 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:50.738243    2553 scope.go:117] "RemoveContainer" containerID="e475807116c278cf76a6a9a8ce18bf9dcac52e6fdb6dc7ddb9c2ec35065e54e3"
Sep 09 16:00:50.749195 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:50.748878    2553 scope.go:117] "RemoveContainer" containerID="e475807116c278cf76a6a9a8ce18bf9dcac52e6fdb6dc7ddb9c2ec35065e54e3"
Sep 09 16:00:50.749335 ip-10-0-129-170 kubenswrapper[2553]: E0909 16:00:50.749313    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e475807116c278cf76a6a9a8ce18bf9dcac52e6fdb6dc7ddb9c2ec35065e54e3\": container with ID starting with e475807116c278cf76a6a9a8ce18bf9dcac52e6fdb6dc7ddb9c2ec35065e54e3 not found: ID does not exist" containerID="e475807116c278cf76a6a9a8ce18bf9dcac52e6fdb6dc7ddb9c2ec35065e54e3"
Sep 09 16:00:50.749382 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:50.749343    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e475807116c278cf76a6a9a8ce18bf9dcac52e6fdb6dc7ddb9c2ec35065e54e3"} err="failed to get container status \"e475807116c278cf76a6a9a8ce18bf9dcac52e6fdb6dc7ddb9c2ec35065e54e3\": rpc error: code = NotFound desc = could not find container \"e475807116c278cf76a6a9a8ce18bf9dcac52e6fdb6dc7ddb9c2ec35065e54e3\": container with ID starting with e475807116c278cf76a6a9a8ce18bf9dcac52e6fdb6dc7ddb9c2ec35065e54e3 not found: ID does not exist"
Sep 09 16:00:50.755070 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:50.755030    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-482dce-5c5c4c564f-vl84m"]
Sep 09 16:00:50.758809 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:50.758783    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-482dce-5c5c4c564f-vl84m"]
Sep 09 16:00:50.770669 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:50.770639    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-482dce-5bf7f8dc94-n77km"]
Sep 09 16:00:50.774448 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:50.774422    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-482dce-5bf7f8dc94-n77km"]
Sep 09 16:00:50.961786 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:50.961684    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5673d1e-abdd-494c-a43b-87e0c0d96f9b" path="/var/lib/kubelet/pods/a5673d1e-abdd-494c-a43b-87e0c0d96f9b/volumes"
Sep 09 16:00:50.962055 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:50.962040    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae3c812e-ebdf-4fc8-81ef-360154273d84" path="/var/lib/kubelet/pods/ae3c812e-ebdf-4fc8-81ef-360154273d84/volumes"
Sep 09 16:00:50.962280 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:50.962268    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8cd335a-399b-4e1f-a515-2628066ccf14" path="/var/lib/kubelet/pods/c8cd335a-399b-4e1f-a515-2628066ccf14/volumes"
Sep 09 16:00:50.962551 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:00:50.962540    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdbe03f8-06de-4b6e-8f5b-960b4ff8a797" path="/var/lib/kubelet/pods/cdbe03f8-06de-4b6e-8f5b-960b4ff8a797/volumes"
Sep 09 16:03:30.962240 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:03:30.962207    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-spk8r"]
Sep 09 16:03:30.962836 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:03:30.962404    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-spk8r" podUID="ea8cd68f-0037-4e71-b169-1dd4bc0ba283" containerName="manager" containerID="cri-o://69ef1d919a2460b60fd9d116bdd2b00b0113cda6daf3ac8168c6ed64b01c7696" gracePeriod=10
Sep 09 16:03:32.541721 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:03:32.541690    2553 generic.go:356] "Generic (PLEG): container finished" podID="ea8cd68f-0037-4e71-b169-1dd4bc0ba283" containerID="69ef1d919a2460b60fd9d116bdd2b00b0113cda6daf3ac8168c6ed64b01c7696" exitCode=0
Sep 09 16:03:32.542155 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:03:32.541778    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-spk8r" event={"ID":"ea8cd68f-0037-4e71-b169-1dd4bc0ba283","Type":"ContainerDied","Data":"69ef1d919a2460b60fd9d116bdd2b00b0113cda6daf3ac8168c6ed64b01c7696"}
Sep 09 16:03:32.611244 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:03:32.611216    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-spk8r"
Sep 09 16:03:32.777066 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:03:32.776972    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea8cd68f-0037-4e71-b169-1dd4bc0ba283-extensions-socket-volume\") pod \"ea8cd68f-0037-4e71-b169-1dd4bc0ba283\" (UID: \"ea8cd68f-0037-4e71-b169-1dd4bc0ba283\") "
Sep 09 16:03:32.777066 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:03:32.777038    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxqkf\" (UniqueName: \"kubernetes.io/projected/ea8cd68f-0037-4e71-b169-1dd4bc0ba283-kube-api-access-nxqkf\") pod \"ea8cd68f-0037-4e71-b169-1dd4bc0ba283\" (UID: \"ea8cd68f-0037-4e71-b169-1dd4bc0ba283\") "
Sep 09 16:03:32.777395 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:03:32.777371    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea8cd68f-0037-4e71-b169-1dd4bc0ba283-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ea8cd68f-0037-4e71-b169-1dd4bc0ba283" (UID: "ea8cd68f-0037-4e71-b169-1dd4bc0ba283"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:03:32.783266 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:03:32.780225    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea8cd68f-0037-4e71-b169-1dd4bc0ba283-kube-api-access-nxqkf" (OuterVolumeSpecName: "kube-api-access-nxqkf") pod "ea8cd68f-0037-4e71-b169-1dd4bc0ba283" (UID: "ea8cd68f-0037-4e71-b169-1dd4bc0ba283"). InnerVolumeSpecName "kube-api-access-nxqkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:03:32.878351 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:03:32.878311    2553 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea8cd68f-0037-4e71-b169-1dd4bc0ba283-extensions-socket-volume\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 16:03:32.878351 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:03:32.878341    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxqkf\" (UniqueName: \"kubernetes.io/projected/ea8cd68f-0037-4e71-b169-1dd4bc0ba283-kube-api-access-nxqkf\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 16:03:33.547928 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:03:33.547898    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-spk8r" event={"ID":"ea8cd68f-0037-4e71-b169-1dd4bc0ba283","Type":"ContainerDied","Data":"751dfdb61fa9b02a3d7ff867c507217f38b3d9d5b2becbceb203dcd7af8e9211"}
Sep 09 16:03:33.548260 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:03:33.547960    2553 scope.go:117] "RemoveContainer" containerID="69ef1d919a2460b60fd9d116bdd2b00b0113cda6daf3ac8168c6ed64b01c7696"
Sep 09 16:03:33.548260 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:03:33.547916    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-spk8r"
Sep 09 16:03:33.566438 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:03:33.566410    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-spk8r"]
Sep 09 16:03:33.571587 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:03:33.571563    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-spk8r"]
Sep 09 16:03:34.961676 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:03:34.961644    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea8cd68f-0037-4e71-b169-1dd4bc0ba283" path="/var/lib/kubelet/pods/ea8cd68f-0037-4e71-b169-1dd4bc0ba283/volumes"
Sep 09 16:04:31.683424 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:04:31.683331    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nsnx9"]
Sep 09 16:04:31.684026 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:04:31.683780    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea8cd68f-0037-4e71-b169-1dd4bc0ba283" containerName="manager"
Sep 09 16:04:31.684026 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:04:31.683802    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea8cd68f-0037-4e71-b169-1dd4bc0ba283" containerName="manager"
Sep 09 16:04:31.684026 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:04:31.683818    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae3c812e-ebdf-4fc8-81ef-360154273d84" containerName="revoke-keys"
Sep 09 16:04:31.684026 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:04:31.683826    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae3c812e-ebdf-4fc8-81ef-360154273d84" containerName="revoke-keys"
Sep 09 16:04:31.684026 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:04:31.683838    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdbe03f8-06de-4b6e-8f5b-960b4ff8a797" containerName="payload-processing"
Sep 09 16:04:31.684026 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:04:31.683846    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdbe03f8-06de-4b6e-8f5b-960b4ff8a797" containerName="payload-processing"
Sep 09 16:04:31.684026 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:04:31.683872    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5673d1e-abdd-494c-a43b-87e0c0d96f9b" containerName="payload-pre-processing"
Sep 09 16:04:31.684026 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:04:31.683881    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5673d1e-abdd-494c-a43b-87e0c0d96f9b" containerName="payload-pre-processing"
Sep 09 16:04:31.684026 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:04:31.683953    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea8cd68f-0037-4e71-b169-1dd4bc0ba283" containerName="manager"
Sep 09 16:04:31.684026 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:04:31.683966    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5673d1e-abdd-494c-a43b-87e0c0d96f9b" containerName="payload-pre-processing"
Sep 09 16:04:31.684026 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:04:31.683979    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdbe03f8-06de-4b6e-8f5b-960b4ff8a797" containerName="payload-processing"
Sep 09 16:04:31.684026 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:04:31.683989    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae3c812e-ebdf-4fc8-81ef-360154273d84" containerName="revoke-keys"
Sep 09 16:04:31.687013 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:04:31.686993    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nsnx9"
Sep 09 16:04:31.689769 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:04:31.689747    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-4zwbk\""
Sep 09 16:04:31.698402 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:04:31.698377    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nsnx9"]
Sep 09 16:04:31.772405 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:04:31.772359    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6af2c5ae-2003-485e-89a8-bf11bf88da38-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-nsnx9\" (UID: \"6af2c5ae-2003-485e-89a8-bf11bf88da38\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nsnx9"
Sep 09 16:04:31.772592 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:04:31.772469    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f759t\" (UniqueName: \"kubernetes.io/projected/6af2c5ae-2003-485e-89a8-bf11bf88da38-kube-api-access-f759t\") pod \"kuadrant-operator-controller-manager-5769f99d9f-nsnx9\" (UID: \"6af2c5ae-2003-485e-89a8-bf11bf88da38\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nsnx9"
Sep 09 16:04:31.873268 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:04:31.873225    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f759t\" (UniqueName: \"kubernetes.io/projected/6af2c5ae-2003-485e-89a8-bf11bf88da38-kube-api-access-f759t\") pod \"kuadrant-operator-controller-manager-5769f99d9f-nsnx9\" (UID: \"6af2c5ae-2003-485e-89a8-bf11bf88da38\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nsnx9"
Sep 09 16:04:31.873453 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:04:31.873303    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6af2c5ae-2003-485e-89a8-bf11bf88da38-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-nsnx9\" (UID: \"6af2c5ae-2003-485e-89a8-bf11bf88da38\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nsnx9"
Sep 09 16:04:31.873658 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:04:31.873639    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6af2c5ae-2003-485e-89a8-bf11bf88da38-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-nsnx9\" (UID: \"6af2c5ae-2003-485e-89a8-bf11bf88da38\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nsnx9"
Sep 09 16:04:31.881868 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:04:31.881841    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f759t\" (UniqueName: \"kubernetes.io/projected/6af2c5ae-2003-485e-89a8-bf11bf88da38-kube-api-access-f759t\") pod \"kuadrant-operator-controller-manager-5769f99d9f-nsnx9\" (UID: \"6af2c5ae-2003-485e-89a8-bf11bf88da38\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nsnx9"
Sep 09 16:04:31.997836 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:04:31.997717    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nsnx9"
Sep 09 16:04:32.278088 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:04:32.278064    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nsnx9"]
Sep 09 16:04:32.280548 ip-10-0-129-170 kubenswrapper[2553]: W0909 16:04:32.280508    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6af2c5ae_2003_485e_89a8_bf11bf88da38.slice/crio-3dccd36e9a6019f55c5585d22e62671ec6047248f5a0d910a7136f7ec0e03319 WatchSource:0}: Error finding container 3dccd36e9a6019f55c5585d22e62671ec6047248f5a0d910a7136f7ec0e03319: Status 404 returned error can't find the container with id 3dccd36e9a6019f55c5585d22e62671ec6047248f5a0d910a7136f7ec0e03319
Sep 09 16:04:32.828960 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:04:32.828921    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nsnx9" event={"ID":"6af2c5ae-2003-485e-89a8-bf11bf88da38","Type":"ContainerStarted","Data":"57f8fc14f882babc2a582fa012317bf17f137c73ca24bff15fe50570ac070051"}
Sep 09 16:04:32.829327 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:04:32.829008    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nsnx9" event={"ID":"6af2c5ae-2003-485e-89a8-bf11bf88da38","Type":"ContainerStarted","Data":"3dccd36e9a6019f55c5585d22e62671ec6047248f5a0d910a7136f7ec0e03319"}
Sep 09 16:04:32.829327 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:04:32.829036    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nsnx9"
Sep 09 16:04:32.854008 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:04:32.853955    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nsnx9" podStartSLOduration=1.85393795 podStartE2EDuration="1.85393795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:04:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:04:32.851701497 +0000 UTC m=+2638.450596143" watchObservedRunningTime="2026-09-09 16:04:32.85393795 +0000 UTC m=+2638.452832594"
Sep 09 16:04:43.837673 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:04:43.837635    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nsnx9"
Sep 09 16:05:19.247745 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:19.247692    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56"]
Sep 09 16:05:19.251266 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:19.251241    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56"
Sep 09 16:05:19.253824 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:19.253800    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-a46aea87-kserve-self-signed-certs\""
Sep 09 16:05:19.260858 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:19.260829    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56"]
Sep 09 16:05:19.291042 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:19.291003    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96a096b5-ca68-4695-9c21-1abc149218d8-home\") pod \"e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56\" (UID: \"96a096b5-ca68-4695-9c21-1abc149218d8\") " pod="llm/e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56"
Sep 09 16:05:19.291245 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:19.291095    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96a096b5-ca68-4695-9c21-1abc149218d8-tls-certs\") pod \"e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56\" (UID: \"96a096b5-ca68-4695-9c21-1abc149218d8\") " pod="llm/e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56"
Sep 09 16:05:19.291245 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:19.291175    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96a096b5-ca68-4695-9c21-1abc149218d8-tmp-dir\") pod \"e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56\" (UID: \"96a096b5-ca68-4695-9c21-1abc149218d8\") " pod="llm/e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56"
Sep 09 16:05:19.291245 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:19.291207    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96a096b5-ca68-4695-9c21-1abc149218d8-dshm\") pod \"e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56\" (UID: \"96a096b5-ca68-4695-9c21-1abc149218d8\") " pod="llm/e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56"
Sep 09 16:05:19.291384 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:19.291244    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96a096b5-ca68-4695-9c21-1abc149218d8-model-cache\") pod \"e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56\" (UID: \"96a096b5-ca68-4695-9c21-1abc149218d8\") " pod="llm/e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56"
Sep 09 16:05:19.291384 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:19.291279    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgwk2\" (UniqueName: \"kubernetes.io/projected/96a096b5-ca68-4695-9c21-1abc149218d8-kube-api-access-lgwk2\") pod \"e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56\" (UID: \"96a096b5-ca68-4695-9c21-1abc149218d8\") " pod="llm/e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56"
Sep 09 16:05:19.392143 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:19.392102    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96a096b5-ca68-4695-9c21-1abc149218d8-home\") pod \"e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56\" (UID: \"96a096b5-ca68-4695-9c21-1abc149218d8\") " pod="llm/e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56"
Sep 09 16:05:19.392347 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:19.392166    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96a096b5-ca68-4695-9c21-1abc149218d8-tls-certs\") pod \"e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56\" (UID: \"96a096b5-ca68-4695-9c21-1abc149218d8\") " pod="llm/e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56"
Sep 09 16:05:19.392347 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:19.392213    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96a096b5-ca68-4695-9c21-1abc149218d8-tmp-dir\") pod \"e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56\" (UID: \"96a096b5-ca68-4695-9c21-1abc149218d8\") " pod="llm/e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56"
Sep 09 16:05:19.392347 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:19.392235    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96a096b5-ca68-4695-9c21-1abc149218d8-dshm\") pod \"e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56\" (UID: \"96a096b5-ca68-4695-9c21-1abc149218d8\") " pod="llm/e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56"
Sep 09 16:05:19.392347 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:19.392279    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96a096b5-ca68-4695-9c21-1abc149218d8-model-cache\") pod \"e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56\" (UID: \"96a096b5-ca68-4695-9c21-1abc149218d8\") " pod="llm/e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56"
Sep 09 16:05:19.392347 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:19.392314    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgwk2\" (UniqueName: \"kubernetes.io/projected/96a096b5-ca68-4695-9c21-1abc149218d8-kube-api-access-lgwk2\") pod \"e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56\" (UID: \"96a096b5-ca68-4695-9c21-1abc149218d8\") " pod="llm/e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56"
Sep 09 16:05:19.392611 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:19.392555    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96a096b5-ca68-4695-9c21-1abc149218d8-home\") pod \"e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56\" (UID: \"96a096b5-ca68-4695-9c21-1abc149218d8\") " pod="llm/e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56"
Sep 09 16:05:19.392611 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:19.392592    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96a096b5-ca68-4695-9c21-1abc149218d8-tmp-dir\") pod \"e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56\" (UID: \"96a096b5-ca68-4695-9c21-1abc149218d8\") " pod="llm/e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56"
Sep 09 16:05:19.392758 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:19.392637    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96a096b5-ca68-4695-9c21-1abc149218d8-model-cache\") pod \"e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56\" (UID: \"96a096b5-ca68-4695-9c21-1abc149218d8\") " pod="llm/e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56"
Sep 09 16:05:19.394518 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:19.394496    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96a096b5-ca68-4695-9c21-1abc149218d8-dshm\") pod \"e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56\" (UID: \"96a096b5-ca68-4695-9c21-1abc149218d8\") " pod="llm/e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56"
Sep 09 16:05:19.394694 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:19.394675    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96a096b5-ca68-4695-9c21-1abc149218d8-tls-certs\") pod \"e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56\" (UID: \"96a096b5-ca68-4695-9c21-1abc149218d8\") " pod="llm/e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56"
Sep 09 16:05:19.401012 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:19.400985    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgwk2\" (UniqueName: \"kubernetes.io/projected/96a096b5-ca68-4695-9c21-1abc149218d8-kube-api-access-lgwk2\") pod \"e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56\" (UID: \"96a096b5-ca68-4695-9c21-1abc149218d8\") " pod="llm/e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56"
Sep 09 16:05:19.561931 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:19.561845    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56"
Sep 09 16:05:19.832787 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:19.832760    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56"]
Sep 09 16:05:19.834959 ip-10-0-129-170 kubenswrapper[2553]: W0909 16:05:19.834932    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96a096b5_ca68_4695_9c21_1abc149218d8.slice/crio-b0c8661f68218488728018a75e9ccea13d0f5aafd9874e7962e16c5632c31a74 WatchSource:0}: Error finding container b0c8661f68218488728018a75e9ccea13d0f5aafd9874e7962e16c5632c31a74: Status 404 returned error can't find the container with id b0c8661f68218488728018a75e9ccea13d0f5aafd9874e7962e16c5632c31a74
Sep 09 16:05:20.054299 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:20.054263    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56" event={"ID":"96a096b5-ca68-4695-9c21-1abc149218d8","Type":"ContainerStarted","Data":"4cd7a0ed976690d00a41d9c2e79b81dff56d10fc9452a4d82962aecb6e769bb9"}
Sep 09 16:05:20.054299 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:20.054304    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56" event={"ID":"96a096b5-ca68-4695-9c21-1abc149218d8","Type":"ContainerStarted","Data":"b0c8661f68218488728018a75e9ccea13d0f5aafd9874e7962e16c5632c31a74"}
Sep 09 16:05:20.070673 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:20.070609    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56" podStartSLOduration=1.070586577 podStartE2EDuration="1.070586577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:05:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:05:20.070206969 +0000 UTC m=+2685.669101614" watchObservedRunningTime="2026-09-09 16:05:20.070586577 +0000 UTC m=+2685.669481226"
Sep 09 16:05:29.562062 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:29.561971    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56"
Sep 09 16:05:29.562062 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:29.562030    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56"
Sep 09 16:05:29.574838 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:29.574814    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56"
Sep 09 16:05:30.115545 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:30.115513    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56"
Sep 09 16:05:35.585963 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:35.585934    2553 scope.go:117] "RemoveContainer" containerID="0bfe74d55edfe5fb31fae0423eec8d98433e4b834d66a8fa4092f83eb8da9d2d"
Sep 09 16:05:35.593894 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:35.593872    2553 scope.go:117] "RemoveContainer" containerID="187bb6e3c90f80ce8c1c6132a64aad0bf54bba97831fd698fc084f88321367a3"
Sep 09 16:05:41.499118 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:41.499072    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56"]
Sep 09 16:05:56.518768 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:56.518679    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56" podUID="96a096b5-ca68-4695-9c21-1abc149218d8" containerName="main" containerID="cri-o://4cd7a0ed976690d00a41d9c2e79b81dff56d10fc9452a4d82962aecb6e769bb9" gracePeriod=45
Sep 09 16:05:56.774104 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:56.774044    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56"
Sep 09 16:05:56.903419 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:56.903384    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgwk2\" (UniqueName: \"kubernetes.io/projected/96a096b5-ca68-4695-9c21-1abc149218d8-kube-api-access-lgwk2\") pod \"96a096b5-ca68-4695-9c21-1abc149218d8\" (UID: \"96a096b5-ca68-4695-9c21-1abc149218d8\") "
Sep 09 16:05:56.903635 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:56.903460    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96a096b5-ca68-4695-9c21-1abc149218d8-dshm\") pod \"96a096b5-ca68-4695-9c21-1abc149218d8\" (UID: \"96a096b5-ca68-4695-9c21-1abc149218d8\") "
Sep 09 16:05:56.903635 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:56.903484    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96a096b5-ca68-4695-9c21-1abc149218d8-tmp-dir\") pod \"96a096b5-ca68-4695-9c21-1abc149218d8\" (UID: \"96a096b5-ca68-4695-9c21-1abc149218d8\") "
Sep 09 16:05:56.903635 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:56.903563    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96a096b5-ca68-4695-9c21-1abc149218d8-model-cache\") pod \"96a096b5-ca68-4695-9c21-1abc149218d8\" (UID: \"96a096b5-ca68-4695-9c21-1abc149218d8\") "
Sep 09 16:05:56.903635 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:56.903594    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96a096b5-ca68-4695-9c21-1abc149218d8-tls-certs\") pod \"96a096b5-ca68-4695-9c21-1abc149218d8\" (UID: \"96a096b5-ca68-4695-9c21-1abc149218d8\") "
Sep 09 16:05:56.903635 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:56.903628    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96a096b5-ca68-4695-9c21-1abc149218d8-home\") pod \"96a096b5-ca68-4695-9c21-1abc149218d8\" (UID: \"96a096b5-ca68-4695-9c21-1abc149218d8\") "
Sep 09 16:05:56.903939 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:56.903840    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a096b5-ca68-4695-9c21-1abc149218d8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "96a096b5-ca68-4695-9c21-1abc149218d8" (UID: "96a096b5-ca68-4695-9c21-1abc149218d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:05:56.903939 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:56.903885    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a096b5-ca68-4695-9c21-1abc149218d8-model-cache" (OuterVolumeSpecName: "model-cache") pod "96a096b5-ca68-4695-9c21-1abc149218d8" (UID: "96a096b5-ca68-4695-9c21-1abc149218d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:05:56.904080 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:56.903933    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96a096b5-ca68-4695-9c21-1abc149218d8-tmp-dir\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 16:05:56.904080 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:56.903984    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a096b5-ca68-4695-9c21-1abc149218d8-home" (OuterVolumeSpecName: "home") pod "96a096b5-ca68-4695-9c21-1abc149218d8" (UID: "96a096b5-ca68-4695-9c21-1abc149218d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:05:56.905766 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:56.905715    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a096b5-ca68-4695-9c21-1abc149218d8-dshm" (OuterVolumeSpecName: "dshm") pod "96a096b5-ca68-4695-9c21-1abc149218d8" (UID: "96a096b5-ca68-4695-9c21-1abc149218d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:05:56.905886 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:56.905826    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a096b5-ca68-4695-9c21-1abc149218d8-kube-api-access-lgwk2" (OuterVolumeSpecName: "kube-api-access-lgwk2") pod "96a096b5-ca68-4695-9c21-1abc149218d8" (UID: "96a096b5-ca68-4695-9c21-1abc149218d8"). InnerVolumeSpecName "kube-api-access-lgwk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:05:56.905886 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:56.905868    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96a096b5-ca68-4695-9c21-1abc149218d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "96a096b5-ca68-4695-9c21-1abc149218d8" (UID: "96a096b5-ca68-4695-9c21-1abc149218d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:05:57.004982 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:57.004944    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96a096b5-ca68-4695-9c21-1abc149218d8-tls-certs\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 16:05:57.004982 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:57.004974    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96a096b5-ca68-4695-9c21-1abc149218d8-home\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 16:05:57.004982 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:57.004986    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgwk2\" (UniqueName: \"kubernetes.io/projected/96a096b5-ca68-4695-9c21-1abc149218d8-kube-api-access-lgwk2\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 16:05:57.005220 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:57.004996    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96a096b5-ca68-4695-9c21-1abc149218d8-dshm\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 16:05:57.005220 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:57.005010    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96a096b5-ca68-4695-9c21-1abc149218d8-model-cache\") on node \"ip-10-0-129-170.ec2.internal\" DevicePath \"\""
Sep 09 16:05:57.233061 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:57.233025    2553 generic.go:356] "Generic (PLEG): container finished" podID="96a096b5-ca68-4695-9c21-1abc149218d8" containerID="4cd7a0ed976690d00a41d9c2e79b81dff56d10fc9452a4d82962aecb6e769bb9" exitCode=0
Sep 09 16:05:57.233248 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:57.233080    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56"
Sep 09 16:05:57.233248 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:57.233102    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56" event={"ID":"96a096b5-ca68-4695-9c21-1abc149218d8","Type":"ContainerDied","Data":"4cd7a0ed976690d00a41d9c2e79b81dff56d10fc9452a4d82962aecb6e769bb9"}
Sep 09 16:05:57.233248 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:57.233145    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56" event={"ID":"96a096b5-ca68-4695-9c21-1abc149218d8","Type":"ContainerDied","Data":"b0c8661f68218488728018a75e9ccea13d0f5aafd9874e7962e16c5632c31a74"}
Sep 09 16:05:57.233248 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:57.233174    2553 scope.go:117] "RemoveContainer" containerID="4cd7a0ed976690d00a41d9c2e79b81dff56d10fc9452a4d82962aecb6e769bb9"
Sep 09 16:05:57.242221 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:57.242200    2553 scope.go:117] "RemoveContainer" containerID="4cd7a0ed976690d00a41d9c2e79b81dff56d10fc9452a4d82962aecb6e769bb9"
Sep 09 16:05:57.242529 ip-10-0-129-170 kubenswrapper[2553]: E0909 16:05:57.242502    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cd7a0ed976690d00a41d9c2e79b81dff56d10fc9452a4d82962aecb6e769bb9\": container with ID starting with 4cd7a0ed976690d00a41d9c2e79b81dff56d10fc9452a4d82962aecb6e769bb9 not found: ID does not exist" containerID="4cd7a0ed976690d00a41d9c2e79b81dff56d10fc9452a4d82962aecb6e769bb9"
Sep 09 16:05:57.242597 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:57.242541    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cd7a0ed976690d00a41d9c2e79b81dff56d10fc9452a4d82962aecb6e769bb9"} err="failed to get container status \"4cd7a0ed976690d00a41d9c2e79b81dff56d10fc9452a4d82962aecb6e769bb9\": rpc error: code = NotFound desc = could not find container \"4cd7a0ed976690d00a41d9c2e79b81dff56d10fc9452a4d82962aecb6e769bb9\": container with ID starting with 4cd7a0ed976690d00a41d9c2e79b81dff56d10fc9452a4d82962aecb6e769bb9 not found: ID does not exist"
Sep 09 16:05:57.254642 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:57.252068    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56"]
Sep 09 16:05:57.255187 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:57.255163    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-a46aea87-kserve-5ccd547659-nlk56"]
Sep 09 16:05:58.962078 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:05:58.962041    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96a096b5-ca68-4695-9c21-1abc149218d8" path="/var/lib/kubelet/pods/96a096b5-ca68-4695-9c21-1abc149218d8/volumes"
Sep 09 16:06:33.295915 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:33.295881    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5d7cdcc7d7-vdtg8_8f94a3e7-a91a-48fc-8f92-0a2e9eaa999e/authorino/0.log"
Sep 09 16:06:35.615699 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:35.615668    2553 scope.go:117] "RemoveContainer" containerID="9ae20dc3416eb852ad1182b7a34500fe123afd9e4eae12dd95499f4dc42e6881"
Sep 09 16:06:35.623720 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:35.623699    2553 scope.go:117] "RemoveContainer" containerID="89a527d5ababa71ad2214178b7a1c7edf9eaa7d1ff23c45200372efdbb8497f9"
Sep 09 16:06:35.631153 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:35.631133    2553 scope.go:117] "RemoveContainer" containerID="232f18da5cee02af50789dc1b47ac11252ad3d9be4b7ca2bf4f46cdd61bddc59"
Sep 09 16:06:35.638186 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:35.638165    2553 scope.go:117] "RemoveContainer" containerID="c6e62d76be344a73eaf75c04123bae588a698b6cf6dbfcab89bc9e07bf77b036"
Sep 09 16:06:39.335940 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:39.335914    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-p5sc6_1a6f2c0c-b12e-4e1a-9192-95987ff1047b/server/0.log"
Sep 09 16:06:39.461384 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:39.461348    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-9nb4x_99e8b784-eaba-41c5-93b1-62683142a4c7/manager/0.log"
Sep 09 16:06:39.717956 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:39.717926    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74fc54996-kqp5k_fb5de8f7-ab6e-426e-9f58-e57de2f6acac/copy-manifests/0.log"
Sep 09 16:06:39.724818 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:39.724791    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74fc54996-kqp5k_fb5de8f7-ab6e-426e-9f58-e57de2f6acac/manager/0.log"
Sep 09 16:06:40.424297 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:40.424265    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29816145-7nbjw_4de61e9c-6f7b-4aec-8793-b679da741a84/cleanup/0.log"
Sep 09 16:06:40.539119 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:40.539081    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29816160-mpgjw_25e8a0ab-9725-47d2-9c06-5cb0db65f98e/cleanup/0.log"
Sep 09 16:06:41.479936 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:41.479902    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5d7cdcc7d7-vdtg8_8f94a3e7-a91a-48fc-8f92-0a2e9eaa999e/authorino/0.log"
Sep 09 16:06:41.847418 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:41.847332    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-wfjvz_5edfa589-8ed0-4951-a1d5-ea7ea08c126c/kuadrant-console-plugin/0.log"
Sep 09 16:06:41.969160 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:41.969129    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-nsnx9_6af2c5ae-2003-485e-89a8-bf11bf88da38/manager/0.log"
Sep 09 16:06:42.223592 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:42.223564    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-7zrwq_e44aa66f-7b4b-48f4-b384-865e874ff99e/manager/0.log"
Sep 09 16:06:42.587395 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:42.587314    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9_9a600f2a-266f-4789-8462-ff24a72f853a/istio-proxy/0.log"
Sep 09 16:06:43.173779 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:43.173725    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-xjtqr_785531c7-305c-49be-b54f-4c0a6d598df3/payload-pre-processing/0.log"
Sep 09 16:06:43.875181 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:43.875151    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-6g8lw_163dac45-745e-4a6e-ac4b-8eb12a7925bb/main/0.log"
Sep 09 16:06:44.450260 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:44.450230    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-nwtk6_c2ec6829-400f-42cc-bb98-f1f084ce16f9/main/0.log"
Sep 09 16:06:50.562893 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:50.562858    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kmsmj/must-gather-j7kr9"]
Sep 09 16:06:50.563283 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:50.563214    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96a096b5-ca68-4695-9c21-1abc149218d8" containerName="main"
Sep 09 16:06:50.563283 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:50.563228    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="96a096b5-ca68-4695-9c21-1abc149218d8" containerName="main"
Sep 09 16:06:50.563283 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:50.563281    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="96a096b5-ca68-4695-9c21-1abc149218d8" containerName="main"
Sep 09 16:06:50.566285 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:50.566266    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kmsmj/must-gather-j7kr9"
Sep 09 16:06:50.568971 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:50.568941    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kmsmj\"/\"openshift-service-ca.crt\""
Sep 09 16:06:50.569563 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:50.569543    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kmsmj\"/\"kube-root-ca.crt\""
Sep 09 16:06:50.570312 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:50.570292    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kmsmj\"/\"default-dockercfg-2c896\""
Sep 09 16:06:50.574420 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:50.574365    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kmsmj/must-gather-j7kr9"]
Sep 09 16:06:50.682160 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:50.682107    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqd62\" (UniqueName: \"kubernetes.io/projected/00127d33-7805-4a5b-b375-13efbdb6d18e-kube-api-access-qqd62\") pod \"must-gather-j7kr9\" (UID: \"00127d33-7805-4a5b-b375-13efbdb6d18e\") " pod="openshift-must-gather-kmsmj/must-gather-j7kr9"
Sep 09 16:06:50.682342 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:50.682226    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/00127d33-7805-4a5b-b375-13efbdb6d18e-must-gather-output\") pod \"must-gather-j7kr9\" (UID: \"00127d33-7805-4a5b-b375-13efbdb6d18e\") " pod="openshift-must-gather-kmsmj/must-gather-j7kr9"
Sep 09 16:06:50.783235 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:50.783194    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqd62\" (UniqueName: \"kubernetes.io/projected/00127d33-7805-4a5b-b375-13efbdb6d18e-kube-api-access-qqd62\") pod \"must-gather-j7kr9\" (UID: \"00127d33-7805-4a5b-b375-13efbdb6d18e\") " pod="openshift-must-gather-kmsmj/must-gather-j7kr9"
Sep 09 16:06:50.783435 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:50.783292    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/00127d33-7805-4a5b-b375-13efbdb6d18e-must-gather-output\") pod \"must-gather-j7kr9\" (UID: \"00127d33-7805-4a5b-b375-13efbdb6d18e\") " pod="openshift-must-gather-kmsmj/must-gather-j7kr9"
Sep 09 16:06:50.783651 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:50.783629    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/00127d33-7805-4a5b-b375-13efbdb6d18e-must-gather-output\") pod \"must-gather-j7kr9\" (UID: \"00127d33-7805-4a5b-b375-13efbdb6d18e\") " pod="openshift-must-gather-kmsmj/must-gather-j7kr9"
Sep 09 16:06:50.793511 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:50.793482    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqd62\" (UniqueName: \"kubernetes.io/projected/00127d33-7805-4a5b-b375-13efbdb6d18e-kube-api-access-qqd62\") pod \"must-gather-j7kr9\" (UID: \"00127d33-7805-4a5b-b375-13efbdb6d18e\") " pod="openshift-must-gather-kmsmj/must-gather-j7kr9"
Sep 09 16:06:50.877174 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:50.877078    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kmsmj/must-gather-j7kr9"
Sep 09 16:06:51.145495 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:51.145460    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kmsmj/must-gather-j7kr9"]
Sep 09 16:06:51.148118 ip-10-0-129-170 kubenswrapper[2553]: W0909 16:06:51.148082    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00127d33_7805_4a5b_b375_13efbdb6d18e.slice/crio-1b3fc6088c0fac1bd8e8a037232f279ee0997833b3e4007a50fd4b9b3d51cfe3 WatchSource:0}: Error finding container 1b3fc6088c0fac1bd8e8a037232f279ee0997833b3e4007a50fd4b9b3d51cfe3: Status 404 returned error can't find the container with id 1b3fc6088c0fac1bd8e8a037232f279ee0997833b3e4007a50fd4b9b3d51cfe3
Sep 09 16:06:51.149905 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:51.149887    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 16:06:51.487143 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:51.487108    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kmsmj/must-gather-j7kr9" event={"ID":"00127d33-7805-4a5b-b375-13efbdb6d18e","Type":"ContainerStarted","Data":"1b3fc6088c0fac1bd8e8a037232f279ee0997833b3e4007a50fd4b9b3d51cfe3"}
Sep 09 16:06:52.496666 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:52.496574    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kmsmj/must-gather-j7kr9" event={"ID":"00127d33-7805-4a5b-b375-13efbdb6d18e","Type":"ContainerStarted","Data":"1796894e43bf236d8598c04866c2f3ba8c5ea15d1aa26f6f7de9bcf28236028a"}
Sep 09 16:06:52.496666 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:52.496626    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kmsmj/must-gather-j7kr9" event={"ID":"00127d33-7805-4a5b-b375-13efbdb6d18e","Type":"ContainerStarted","Data":"f7645f0251a32e3e9a084cbbcb7ff39c623c615b276fbbe80fdb2527b7f53665"}
Sep 09 16:06:52.513618 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:52.513552    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kmsmj/must-gather-j7kr9" podStartSLOduration=1.577648286 podStartE2EDuration="2.513533817s" totalImagesPullingTime="935.885531ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:06:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:06:52.511959029 +0000 UTC m=+2778.110853686" watchObservedRunningTime="2026-09-09 16:06:52.513533817 +0000 UTC m=+2778.112428461"
Sep 09 16:06:53.691223 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:53.691186    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kcl4l_26591625-285a-4c93-b5be-7d0f4f0a0aae/global-pull-secret-syncer/0.log"
Sep 09 16:06:53.840173 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:53.840141    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xjfqb_3a444239-19ad-490a-a006-0a43ce90c9ef/konnectivity-agent/0.log"
Sep 09 16:06:53.859299 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:53.859273    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-170.ec2.internal_f934fa7e403724f763f7c170a46e1bd9/haproxy/0.log"
Sep 09 16:06:58.045946 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:58.045849    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5d7cdcc7d7-vdtg8_8f94a3e7-a91a-48fc-8f92-0a2e9eaa999e/authorino/0.log"
Sep 09 16:06:58.160629 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:58.160589    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-wfjvz_5edfa589-8ed0-4951-a1d5-ea7ea08c126c/kuadrant-console-plugin/0.log"
Sep 09 16:06:58.218877 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:58.218841    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-nsnx9_6af2c5ae-2003-485e-89a8-bf11bf88da38/manager/0.log"
Sep 09 16:06:58.330789 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:06:58.330679    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-7zrwq_e44aa66f-7b4b-48f4-b384-865e874ff99e/manager/0.log"
Sep 09 16:07:00.704716 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:00.704634    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n9qh8_6654b5d1-f133-49eb-9062-74548d0e0670/node-exporter/0.log"
Sep 09 16:07:00.729257 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:00.729227    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n9qh8_6654b5d1-f133-49eb-9062-74548d0e0670/kube-rbac-proxy/0.log"
Sep 09 16:07:00.751209 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:00.751183    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n9qh8_6654b5d1-f133-49eb-9062-74548d0e0670/init-textfile/0.log"
Sep 09 16:07:04.522831 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:04.522791    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kmsmj/perf-node-gather-daemonset-m4xrb"]
Sep 09 16:07:04.529526 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:04.529286    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-m4xrb"
Sep 09 16:07:04.536836 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:04.536808    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kmsmj/perf-node-gather-daemonset-m4xrb"]
Sep 09 16:07:04.640761 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:04.639256    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/78adb1be-edaf-4197-a9f9-38e6516ed3e6-proc\") pod \"perf-node-gather-daemonset-m4xrb\" (UID: \"78adb1be-edaf-4197-a9f9-38e6516ed3e6\") " pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-m4xrb"
Sep 09 16:07:04.640761 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:04.639370    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/78adb1be-edaf-4197-a9f9-38e6516ed3e6-podres\") pod \"perf-node-gather-daemonset-m4xrb\" (UID: \"78adb1be-edaf-4197-a9f9-38e6516ed3e6\") " pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-m4xrb"
Sep 09 16:07:04.640761 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:04.639398    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g56ng\" (UniqueName: \"kubernetes.io/projected/78adb1be-edaf-4197-a9f9-38e6516ed3e6-kube-api-access-g56ng\") pod \"perf-node-gather-daemonset-m4xrb\" (UID: \"78adb1be-edaf-4197-a9f9-38e6516ed3e6\") " pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-m4xrb"
Sep 09 16:07:04.640761 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:04.639437    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/78adb1be-edaf-4197-a9f9-38e6516ed3e6-lib-modules\") pod \"perf-node-gather-daemonset-m4xrb\" (UID: \"78adb1be-edaf-4197-a9f9-38e6516ed3e6\") " pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-m4xrb"
Sep 09 16:07:04.640761 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:04.639493    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78adb1be-edaf-4197-a9f9-38e6516ed3e6-sys\") pod \"perf-node-gather-daemonset-m4xrb\" (UID: \"78adb1be-edaf-4197-a9f9-38e6516ed3e6\") " pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-m4xrb"
Sep 09 16:07:04.740244 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:04.740167    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/78adb1be-edaf-4197-a9f9-38e6516ed3e6-proc\") pod \"perf-node-gather-daemonset-m4xrb\" (UID: \"78adb1be-edaf-4197-a9f9-38e6516ed3e6\") " pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-m4xrb"
Sep 09 16:07:04.740447 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:04.740334    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/78adb1be-edaf-4197-a9f9-38e6516ed3e6-podres\") pod \"perf-node-gather-daemonset-m4xrb\" (UID: \"78adb1be-edaf-4197-a9f9-38e6516ed3e6\") " pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-m4xrb"
Sep 09 16:07:04.740447 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:04.740363    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g56ng\" (UniqueName: \"kubernetes.io/projected/78adb1be-edaf-4197-a9f9-38e6516ed3e6-kube-api-access-g56ng\") pod \"perf-node-gather-daemonset-m4xrb\" (UID: \"78adb1be-edaf-4197-a9f9-38e6516ed3e6\") " pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-m4xrb"
Sep 09 16:07:04.740447 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:04.740403    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/78adb1be-edaf-4197-a9f9-38e6516ed3e6-lib-modules\") pod \"perf-node-gather-daemonset-m4xrb\" (UID: \"78adb1be-edaf-4197-a9f9-38e6516ed3e6\") " pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-m4xrb"
Sep 09 16:07:04.740635 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:04.740464    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78adb1be-edaf-4197-a9f9-38e6516ed3e6-sys\") pod \"perf-node-gather-daemonset-m4xrb\" (UID: \"78adb1be-edaf-4197-a9f9-38e6516ed3e6\") " pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-m4xrb"
Sep 09 16:07:04.740635 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:04.740565    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78adb1be-edaf-4197-a9f9-38e6516ed3e6-sys\") pod \"perf-node-gather-daemonset-m4xrb\" (UID: \"78adb1be-edaf-4197-a9f9-38e6516ed3e6\") " pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-m4xrb"
Sep 09 16:07:04.740635 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:04.740627    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/78adb1be-edaf-4197-a9f9-38e6516ed3e6-proc\") pod \"perf-node-gather-daemonset-m4xrb\" (UID: \"78adb1be-edaf-4197-a9f9-38e6516ed3e6\") " pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-m4xrb"
Sep 09 16:07:04.740903 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:04.740781    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/78adb1be-edaf-4197-a9f9-38e6516ed3e6-podres\") pod \"perf-node-gather-daemonset-m4xrb\" (UID: \"78adb1be-edaf-4197-a9f9-38e6516ed3e6\") " pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-m4xrb"
Sep 09 16:07:04.741228 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:04.741203    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/78adb1be-edaf-4197-a9f9-38e6516ed3e6-lib-modules\") pod \"perf-node-gather-daemonset-m4xrb\" (UID: \"78adb1be-edaf-4197-a9f9-38e6516ed3e6\") " pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-m4xrb"
Sep 09 16:07:04.748805 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:04.748769    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g56ng\" (UniqueName: \"kubernetes.io/projected/78adb1be-edaf-4197-a9f9-38e6516ed3e6-kube-api-access-g56ng\") pod \"perf-node-gather-daemonset-m4xrb\" (UID: \"78adb1be-edaf-4197-a9f9-38e6516ed3e6\") " pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-m4xrb"
Sep 09 16:07:04.845927 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:04.845828    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-m4xrb"
Sep 09 16:07:05.141276 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:05.141245    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kmsmj/perf-node-gather-daemonset-m4xrb"]
Sep 09 16:07:05.146174 ip-10-0-129-170 kubenswrapper[2553]: W0909 16:07:05.146137    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod78adb1be_edaf_4197_a9f9_38e6516ed3e6.slice/crio-97ec3e1c758cb8840295f8372d76c9364294bb3e66a6fd41d9734efd40ccd501 WatchSource:0}: Error finding container 97ec3e1c758cb8840295f8372d76c9364294bb3e66a6fd41d9734efd40ccd501: Status 404 returned error can't find the container with id 97ec3e1c758cb8840295f8372d76c9364294bb3e66a6fd41d9734efd40ccd501
Sep 09 16:07:05.594269 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:05.594233    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-m4xrb" event={"ID":"78adb1be-edaf-4197-a9f9-38e6516ed3e6","Type":"ContainerStarted","Data":"28b28920a6cdab43db6a6db03efec28001b2a36fe6d57778bf971d2270cbf30a"}
Sep 09 16:07:05.594864 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:05.594837    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-m4xrb" event={"ID":"78adb1be-edaf-4197-a9f9-38e6516ed3e6","Type":"ContainerStarted","Data":"97ec3e1c758cb8840295f8372d76c9364294bb3e66a6fd41d9734efd40ccd501"}
Sep 09 16:07:05.595018 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:05.595005    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-m4xrb"
Sep 09 16:07:05.611454 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:05.611394    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-m4xrb" podStartSLOduration=1.611373942 podStartE2EDuration="1.611373942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:07:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:07:05.610979719 +0000 UTC m=+2791.209874365" watchObservedRunningTime="2026-09-09 16:07:05.611373942 +0000 UTC m=+2791.210268586"
Sep 09 16:07:06.291176 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:06.291147    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cwkld_acc8de36-2e42-4471-a31e-38c66f87ab8c/dns/0.log"
Sep 09 16:07:06.313435 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:06.313406    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cwkld_acc8de36-2e42-4471-a31e-38c66f87ab8c/kube-rbac-proxy/0.log"
Sep 09 16:07:06.474609 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:06.474573    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wgm9j_3cfc8b11-7c62-46fc-8a5e-d5d889501cfd/dns-node-resolver/0.log"
Sep 09 16:07:07.020959 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:07.020924    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-d78b99cdd-bhxqb_d297493c-df72-413e-988f-c4d38b6bbf5f/registry/0.log"
Sep 09 16:07:07.092525 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:07.092497    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-txxcj_1aaf71f6-9195-4b50-a9a4-3cb85b10956a/node-ca/0.log"
Sep 09 16:07:07.981999 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:07.981960    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9_9a600f2a-266f-4789-8462-ff24a72f853a/istio-proxy/0.log"
Sep 09 16:07:08.423831 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:08.423796    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-xjtqr_785531c7-305c-49be-b54f-4c0a6d598df3/payload-pre-processing/0.log"
Sep 09 16:07:09.370574 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:09.370546    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7kz9s_d2082125-0643-4b68-a968-a45d7d8aac36/serve-healthcheck-canary/0.log"
Sep 09 16:07:09.939965 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:09.939925    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hz29k_f3f904f8-3a34-41d2-8f0c-0695ad8b5209/kube-rbac-proxy/0.log"
Sep 09 16:07:09.960602 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:09.960576    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hz29k_f3f904f8-3a34-41d2-8f0c-0695ad8b5209/exporter/0.log"
Sep 09 16:07:09.982151 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:09.982120    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hz29k_f3f904f8-3a34-41d2-8f0c-0695ad8b5209/extractor/0.log"
Sep 09 16:07:12.618706 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:12.618670    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-m4xrb"
Sep 09 16:07:12.634706 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:12.634674    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-p5sc6_1a6f2c0c-b12e-4e1a-9192-95987ff1047b/server/0.log"
Sep 09 16:07:12.754166 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:12.754134    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-9nb4x_99e8b784-eaba-41c5-93b1-62683142a4c7/manager/0.log"
Sep 09 16:07:12.864097 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:12.864035    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74fc54996-kqp5k_fb5de8f7-ab6e-426e-9f58-e57de2f6acac/manager/0.log"
Sep 09 16:07:12.884348 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:12.884259    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74fc54996-kqp5k_fb5de8f7-ab6e-426e-9f58-e57de2f6acac/copy-manifests/0.log"
Sep 09 16:07:14.348261 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:14.348218    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-76f449f6db-2j5kb_597e9f25-fee1-4d02-b59f-695e293df055/manager/0.log"
Sep 09 16:07:14.403441 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:14.403345    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-mnnvk_b81841cc-d69c-4568-99af-252d0c1f8996/openshift-lws-operator/0.log"
Sep 09 16:07:14.811083 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:14.811033    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9_9a600f2a-266f-4789-8462-ff24a72f853a/istio-proxy/0.log"
Sep 09 16:07:15.181046 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:15.181001    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-xjtqr_785531c7-305c-49be-b54f-4c0a6d598df3/payload-pre-processing/0.log"
Sep 09 16:07:16.817131 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:16.817099    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-nwks9_9a600f2a-266f-4789-8462-ff24a72f853a/istio-proxy/0.log"
Sep 09 16:07:17.209544 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:17.209503    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-xjtqr_785531c7-305c-49be-b54f-4c0a6d598df3/payload-pre-processing/0.log"
Sep 09 16:07:21.157309 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:21.157280    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cmk65_4a532bf5-55b6-46a3-8c3f-4f3af4920a8d/kube-multus-additional-cni-plugins/0.log"
Sep 09 16:07:21.176947 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:21.176897    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cmk65_4a532bf5-55b6-46a3-8c3f-4f3af4920a8d/egress-router-binary-copy/0.log"
Sep 09 16:07:21.196598 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:21.196552    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cmk65_4a532bf5-55b6-46a3-8c3f-4f3af4920a8d/cni-plugins/0.log"
Sep 09 16:07:21.216191 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:21.216161    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cmk65_4a532bf5-55b6-46a3-8c3f-4f3af4920a8d/bond-cni-plugin/0.log"
Sep 09 16:07:21.235002 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:21.234977    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cmk65_4a532bf5-55b6-46a3-8c3f-4f3af4920a8d/routeoverride-cni/0.log"
Sep 09 16:07:21.255403 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:21.255378    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cmk65_4a532bf5-55b6-46a3-8c3f-4f3af4920a8d/whereabouts-cni-bincopy/0.log"
Sep 09 16:07:21.274862 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:21.274831    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cmk65_4a532bf5-55b6-46a3-8c3f-4f3af4920a8d/whereabouts-cni/0.log"
Sep 09 16:07:21.475874 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:21.475850    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-bt4f2_d1a7a818-24f1-4d7c-a3d5-416915c6c2cf/kube-multus/0.log"
Sep 09 16:07:21.628339 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:21.628300    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-v5kbj_57473bc0-f6d0-45f1-8954-93a7d2f4321b/network-metrics-daemon/0.log"
Sep 09 16:07:21.644015 ip-10-0-129-170 kubenswrapper[2553]: I0909 16:07:21.643989    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-v5kbj_57473bc0-f6d0-45f1-8954-93a7d2f4321b/kube-rbac-proxy/0.log"