Sep 16 12:36:18.298526 ip-10-0-133-190 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 16 12:36:18.298537 ip-10-0-133-190 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 16 12:36:18.298545 ip-10-0-133-190 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 16 12:36:18.298784 ip-10-0-133-190 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 16 12:36:28.113460 ip-10-0-133-190 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 16 12:36:28.113475 ip-10-0-133-190 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot ae35a1419d024c299561462813303c48 --
Sep 16 12:38:13.426326 ip-10-0-133-190 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 12:38:13.882734 ip-10-0-133-190 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 12:38:13.882734 ip-10-0-133-190 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 12:38:13.882734 ip-10-0-133-190 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 12:38:13.882734 ip-10-0-133-190 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 12:38:13.882734 ip-10-0-133-190 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 12:38:13.885379 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.885300    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 12:38:13.889320 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889305    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:38:13.889320 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889320    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:38:13.889387 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889324    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:38:13.889387 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889327    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:38:13.889387 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889330    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:38:13.889387 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889333    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:38:13.889387 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889336    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:38:13.889387 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889339    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:38:13.889387 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889341    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:38:13.889387 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889344    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:38:13.889387 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889347    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:38:13.889387 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889365    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:38:13.889387 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889368    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:38:13.889387 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889371    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:38:13.889387 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889373    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:38:13.889387 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889376    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:38:13.889387 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889380    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:38:13.889387 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889384    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:38:13.889387 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889387    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:38:13.889387 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889390    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:38:13.889387 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889393    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:38:13.889387 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889396    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:38:13.889868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889399    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:38:13.889868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889402    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:38:13.889868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889404    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:38:13.889868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889407    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:38:13.889868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889409    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:38:13.889868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889412    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:38:13.889868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889414    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:38:13.889868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889416    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:38:13.889868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889419    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:38:13.889868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889422    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:38:13.889868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889424    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:38:13.889868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889427    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:38:13.889868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889429    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:38:13.889868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889432    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:38:13.889868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889434    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:38:13.889868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889436    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:38:13.889868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889439    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:38:13.889868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889442    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:38:13.889868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889444    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:38:13.889868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889446    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:38:13.890407 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889449    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:38:13.890407 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889451    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:38:13.890407 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889453    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:38:13.890407 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889457    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:38:13.890407 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889460    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:38:13.890407 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889464    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:38:13.890407 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889466    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:38:13.890407 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889469    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:38:13.890407 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889471    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:38:13.890407 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889474    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:38:13.890407 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889478    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:38:13.890407 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889481    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:38:13.890407 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889485    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:38:13.890407 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889488    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:38:13.890407 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889491    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:38:13.890407 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889493    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:38:13.890407 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889496    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:38:13.890407 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889499    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:38:13.890407 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889501    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:38:13.890883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889504    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:38:13.890883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889506    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:38:13.890883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889509    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:38:13.890883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889511    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:38:13.890883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889513    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:38:13.890883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889516    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:38:13.890883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889518    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:38:13.890883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889521    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:38:13.890883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889523    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:38:13.890883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889526    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:38:13.890883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889529    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:38:13.890883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889531    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:38:13.890883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889534    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:38:13.890883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889536    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:38:13.890883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889539    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:38:13.890883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889541    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:38:13.890883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889544    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:38:13.890883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889546    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:38:13.890883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889548    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:38:13.890883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889551    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:38:13.891387 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889553    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:38:13.891387 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889556    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:38:13.891387 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889558    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:38:13.891387 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889560    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:38:13.891387 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889563    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:38:13.891387 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889565    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:38:13.891387 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889568    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:38:13.891387 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889571    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:38:13.891387 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889573    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:38:13.891387 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889576    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:38:13.891387 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889579    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:38:13.891387 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889581    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:38:13.891387 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889583    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:38:13.891387 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889586    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:38:13.891387 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889588    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:38:13.891387 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889591    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:38:13.891387 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889593    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:38:13.891387 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.889596    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:38:13.891387 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890277    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:38:13.891883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890284    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:38:13.891883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890286    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:38:13.891883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890289    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:38:13.891883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890292    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:38:13.891883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890295    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:38:13.891883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890298    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:38:13.891883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890300    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:38:13.891883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890304    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:38:13.891883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890307    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:38:13.891883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890310    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:38:13.891883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890312    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:38:13.891883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890315    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:38:13.891883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890317    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:38:13.891883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890320    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:38:13.891883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890323    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:38:13.891883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890325    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:38:13.891883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890327    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:38:13.891883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890330    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:38:13.891883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890333    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:38:13.891883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890335    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:38:13.892407 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890338    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:38:13.892407 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890341    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:38:13.892407 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890343    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:38:13.892407 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890345    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:38:13.892407 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890349    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:38:13.892407 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890353    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:38:13.892407 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890356    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:38:13.892407 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890359    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:38:13.892407 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890361    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:38:13.892407 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890363    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:38:13.892407 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890366    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:38:13.892407 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890369    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:38:13.892407 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890371    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:38:13.892407 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890373    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:38:13.892407 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890376    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:38:13.892407 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890378    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:38:13.892407 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890381    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:38:13.892407 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890383    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:38:13.892407 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890386    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:38:13.892407 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890389    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:38:13.892938 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890392    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:38:13.892938 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890394    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:38:13.892938 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890397    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:38:13.892938 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890399    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:38:13.892938 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890402    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:38:13.892938 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890405    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:38:13.892938 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890407    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:38:13.892938 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890410    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:38:13.892938 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890412    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:38:13.892938 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890416    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:38:13.892938 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890419    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:38:13.892938 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890421    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:38:13.892938 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890424    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:38:13.892938 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890428    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:38:13.892938 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890432    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:38:13.892938 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890434    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:38:13.892938 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890437    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:38:13.892938 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890440    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:38:13.892938 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890443    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:38:13.892938 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890446    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:38:13.893443 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890448    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:38:13.893443 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890451    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:38:13.893443 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890453    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:38:13.893443 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890456    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:38:13.893443 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890458    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:38:13.893443 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890460    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:38:13.893443 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890464    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:38:13.893443 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890467    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:38:13.893443 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890470    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:38:13.893443 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890473    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:38:13.893443 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890477    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:38:13.893443 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890481    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:38:13.893443 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890484    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:38:13.893443 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890486    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:38:13.893443 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890489    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:38:13.893443 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890492    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:38:13.893443 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890494    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:38:13.893443 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890497    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:38:13.893443 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890499    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:38:13.893911 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890502    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:38:13.893911 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890504    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:38:13.893911 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890506    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:38:13.893911 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890509    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:38:13.893911 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890511    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:38:13.893911 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890514    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:38:13.893911 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890516    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:38:13.893911 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890518    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:38:13.893911 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890521    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:38:13.893911 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890523    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:38:13.893911 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890525    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:38:13.893911 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890528    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:38:13.893911 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890530    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:38:13.893911 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890532    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:38:13.893911 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890535    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:38:13.893911 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890537    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:38:13.893911 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890540    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:38:13.893911 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890542    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:38:13.893911 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890544    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:38:13.893911 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890908    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:38:13.894410 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890913    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:38:13.894410 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890916    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:38:13.894410 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890919    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:38:13.894410 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890926    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:38:13.894410 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890929    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:38:13.894410 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890932    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:38:13.894410 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890934    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:38:13.894410 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890937    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:38:13.894410 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890940    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:38:13.894410 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890944    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:38:13.894410 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890947    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:38:13.894410 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890951    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:38:13.894410 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890953    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:38:13.894410 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890956    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:38:13.894410 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890959    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:38:13.894410 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890961    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:38:13.894410 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890964    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:38:13.894410 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890966    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:38:13.894410 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890969    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:38:13.894901 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890971    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:38:13.894901 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890974    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:38:13.894901 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890976    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:38:13.894901 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890978    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:38:13.894901 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890981    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:38:13.894901 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890983    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:38:13.894901 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890986    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:38:13.894901 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890988    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:38:13.894901 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890991    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:38:13.894901 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890994    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:38:13.894901 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.890997    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:38:13.894901 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891000    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:38:13.894901 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891003    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:38:13.894901 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891005    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:38:13.894901 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891009    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:38:13.894901 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891013    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:38:13.894901 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891016    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:38:13.894901 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891020    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:38:13.894901 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891040    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:38:13.895399 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891043    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:38:13.895399 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891045    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:38:13.895399 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891048    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:38:13.895399 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891051    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:38:13.895399 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891053    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:38:13.895399 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891056    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:38:13.895399 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891058    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:38:13.895399 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891061    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:38:13.895399 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891064    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:38:13.895399 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891066    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:38:13.895399 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891069    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:38:13.895399 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891071    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:38:13.895399 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891074    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:38:13.895399 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891076    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:38:13.895399 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891079    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:38:13.895399 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891081    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:38:13.895399 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891084    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:38:13.895399 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891086    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:38:13.895399 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891089    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:38:13.895399 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891092    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:38:13.895868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891094    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:38:13.895868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891097    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:38:13.895868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891100    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:38:13.895868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891102    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:38:13.895868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891105    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:38:13.895868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891107    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:38:13.895868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891110    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:38:13.895868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891112    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:38:13.895868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891114    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:38:13.895868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891121    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:38:13.895868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891123    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:38:13.895868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891126    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:38:13.895868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891129    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:38:13.895868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891131    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:38:13.895868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891134    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:38:13.895868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891136    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:38:13.895868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891139    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:38:13.895868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891141    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:38:13.895868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891145    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:38:13.895868 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891149    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:38:13.896376 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891151    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:38:13.896376 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891154    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:38:13.896376 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891156    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:38:13.896376 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891158    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:38:13.896376 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891161    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:38:13.896376 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891163    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:38:13.896376 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891166    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:38:13.896376 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891168    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:38:13.896376 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891171    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:38:13.896376 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891173    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:38:13.896376 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891176    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:38:13.896376 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891178    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:38:13.896376 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891181    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:38:13.896376 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891184    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:38:13.896376 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891186    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:38:13.896376 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891189    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:38:13.896376 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891191    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:38:13.896376 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891194    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:38:13.896376 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891196    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:38:13.896376 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891199    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:38:13.896863 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891269    2577 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 12:38:13.896863 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891308    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 12:38:13.896863 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891316    2577 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 12:38:13.896863 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891320    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 12:38:13.896863 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891325    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 12:38:13.896863 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891328    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 12:38:13.896863 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891332    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 12:38:13.896863 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891336    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 12:38:13.896863 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891339    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 12:38:13.896863 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891342    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 12:38:13.896863 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891345    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 12:38:13.896863 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891348    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 12:38:13.896863 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891351    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 12:38:13.896863 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891354    2577 flags.go:64] FLAG: --cgroup-root=""
Sep 16 12:38:13.896863 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891357    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 12:38:13.896863 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891360    2577 flags.go:64] FLAG: --client-ca-file=""
Sep 16 12:38:13.896863 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891363    2577 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 12:38:13.896863 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891366    2577 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 12:38:13.896863 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891370    2577 flags.go:64] FLAG: --cluster-domain=""
Sep 16 12:38:13.896863 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891373    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 12:38:13.896863 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891376    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 12:38:13.896863 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891379    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 12:38:13.896863 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891382    2577 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 12:38:13.896863 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891385    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 12:38:13.897471 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891388    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 12:38:13.897471 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891392    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 12:38:13.897471 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891395    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 12:38:13.897471 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891398    2577 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 12:38:13.897471 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891401    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 12:38:13.897471 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891404    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 12:38:13.897471 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891407    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 12:38:13.897471 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891410    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 12:38:13.897471 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891414    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 12:38:13.897471 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891418    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 12:38:13.897471 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891421    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 12:38:13.897471 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891424    2577 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 12:38:13.897471 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891426    2577 flags.go:64] FLAG: --enable-server="true"
Sep 16 12:38:13.897471 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891430    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 12:38:13.897471 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891434    2577 flags.go:64] FLAG: --event-burst="100"
Sep 16 12:38:13.897471 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891437    2577 flags.go:64] FLAG: --event-qps="50"
Sep 16 12:38:13.897471 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891440    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 12:38:13.897471 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891443    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 12:38:13.897471 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891445    2577 flags.go:64] FLAG: --eviction-hard=""
Sep 16 12:38:13.897471 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891449    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 12:38:13.897471 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891452    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 12:38:13.897471 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891455    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 12:38:13.897471 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891457    2577 flags.go:64] FLAG: --eviction-soft=""
Sep 16 12:38:13.897471 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891460    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 12:38:13.897471 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891463    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 12:38:13.898137 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891465    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 12:38:13.898137 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891468    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 12:38:13.898137 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891471    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 12:38:13.898137 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891473    2577 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 12:38:13.898137 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891476    2577 flags.go:64] FLAG: --feature-gates=""
Sep 16 12:38:13.898137 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891479    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 12:38:13.898137 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891482    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 12:38:13.898137 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891485    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 12:38:13.898137 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891488    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 12:38:13.898137 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891491    2577 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 12:38:13.898137 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891494    2577 flags.go:64] FLAG: --help="false"
Sep 16 12:38:13.898137 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891497    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-133-190.ec2.internal"
Sep 16 12:38:13.898137 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891500    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 12:38:13.898137 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891502    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 12:38:13.898137 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891505    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 12:38:13.898137 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891508    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 12:38:13.898137 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891513    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 12:38:13.898137 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891515    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 12:38:13.898137 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891518    2577 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 12:38:13.898137 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891521    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 12:38:13.898137 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891523    2577 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 12:38:13.898137 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891527    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 12:38:13.898137 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891530    2577 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 12:38:13.898137 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891533    2577 flags.go:64] FLAG: --kube-reserved=""
Sep 16 12:38:13.898741 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891536    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 12:38:13.898741 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891539    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 12:38:13.898741 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891542    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 12:38:13.898741 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891544    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 12:38:13.898741 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891547    2577 flags.go:64] FLAG: --lock-file=""
Sep 16 12:38:13.898741 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891550    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 12:38:13.898741 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891552    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 12:38:13.898741 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891555    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 12:38:13.898741 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891560    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 12:38:13.898741 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891563    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 12:38:13.898741 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891566    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 12:38:13.898741 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891568    2577 flags.go:64] FLAG: --logging-format="text"
Sep 16 12:38:13.898741 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891571    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 12:38:13.898741 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891574    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 12:38:13.898741 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891577    2577 flags.go:64] FLAG: --manifest-url=""
Sep 16 12:38:13.898741 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891579    2577 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 12:38:13.898741 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891590    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 12:38:13.898741 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891593    2577 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 12:38:13.898741 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891597    2577 flags.go:64] FLAG: --max-pods="110"
Sep 16 12:38:13.898741 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891599    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 12:38:13.898741 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891602    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 12:38:13.898741 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891605    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 12:38:13.898741 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891608    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 12:38:13.898741 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891610    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 12:38:13.898741 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891614    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 12:38:13.899416 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891617    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 12:38:13.899416 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891624    2577 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 12:38:13.899416 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891627    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 12:38:13.899416 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891630    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 12:38:13.899416 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891633    2577 flags.go:64] FLAG: --pod-cidr=""
Sep 16 12:38:13.899416 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891636    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 12:38:13.899416 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891642    2577 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 12:38:13.899416 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891645    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 12:38:13.899416 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891648    2577 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 12:38:13.899416 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891650    2577 flags.go:64] FLAG: --port="10250"
Sep 16 12:38:13.899416 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891653    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 12:38:13.899416 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891656    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-011e0c78667dbcb16"
Sep 16 12:38:13.899416 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891659    2577 flags.go:64] FLAG: --qos-reserved=""
Sep 16 12:38:13.899416 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891662    2577 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 12:38:13.899416 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891664    2577 flags.go:64] FLAG: --register-node="true"
Sep 16 12:38:13.899416 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891667    2577 flags.go:64] FLAG: --register-with-taints=""
Sep 16 12:38:13.899416 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891670    2577 flags.go:64] FLAG: --registry-burst="10"
Sep 16 12:38:13.899416 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891673    2577 flags.go:64] FLAG: --registry-qps="5"
Sep 16 12:38:13.899416 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891676    2577 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 12:38:13.899416 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891678    2577 flags.go:64] FLAG: --reserved-memory=""
Sep 16 12:38:13.899416 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891681    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 12:38:13.899416 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891684    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 12:38:13.899416 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891687    2577 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 12:38:13.899416 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891691    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 12:38:13.899416 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891694    2577 flags.go:64] FLAG: --runonce="false"
Sep 16 12:38:13.900069 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891697    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 12:38:13.900069 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891700    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 12:38:13.900069 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891702    2577 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 12:38:13.900069 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891705    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 12:38:13.900069 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891708    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 12:38:13.900069 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891711    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 12:38:13.900069 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891714    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 12:38:13.900069 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891718    2577 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 12:38:13.900069 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891721    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 12:38:13.900069 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891725    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 12:38:13.900069 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891727    2577 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 12:38:13.900069 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891730    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 12:38:13.900069 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891733    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 12:38:13.900069 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891737    2577 flags.go:64] FLAG: --system-cgroups=""
Sep 16 12:38:13.900069 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891739    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 12:38:13.900069 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891744    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 12:38:13.900069 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891747    2577 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 12:38:13.900069 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891750    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 12:38:13.900069 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891754    2577 flags.go:64] FLAG: --tls-min-version=""
Sep 16 12:38:13.900069 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891757    2577 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 12:38:13.900069 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891760    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 12:38:13.900069 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891763    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 12:38:13.900069 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891765    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 12:38:13.900069 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891768    2577 flags.go:64] FLAG: --v="2"
Sep 16 12:38:13.900069 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891772    2577 flags.go:64] FLAG: --version="false"
Sep 16 12:38:13.900675 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891776    2577 flags.go:64] FLAG: --vmodule=""
Sep 16 12:38:13.900675 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891779    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 12:38:13.900675 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.891782    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 12:38:13.900675 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891907    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:38:13.900675 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891912    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:38:13.900675 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891917    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:38:13.900675 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891920    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:38:13.900675 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891923    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:38:13.900675 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891926    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:38:13.900675 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891929    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:38:13.900675 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891931    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:38:13.900675 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891934    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:38:13.900675 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891936    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:38:13.900675 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891939    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:38:13.900675 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891942    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:38:13.900675 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891945    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:38:13.900675 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891947    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:38:13.900675 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891950    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:38:13.900675 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891952    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:38:13.900675 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891955    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:38:13.901189 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891973    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:38:13.901189 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891977    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:38:13.901189 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891980    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:38:13.901189 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891983    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:38:13.901189 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891986    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:38:13.901189 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891989    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:38:13.901189 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891991    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:38:13.901189 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891994    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:38:13.901189 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.891997    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:38:13.901189 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892001    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:38:13.901189 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892004    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:38:13.901189 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892007    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:38:13.901189 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892010    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:38:13.901189 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892013    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:38:13.901189 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892015    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:38:13.901189 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892018    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:38:13.901189 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892021    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:38:13.901189 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892040    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:38:13.901189 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892043    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:38:13.901189 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892046    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:38:13.901680 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892048    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:38:13.901680 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892051    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:38:13.901680 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892054    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:38:13.901680 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892057    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:38:13.901680 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892059    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:38:13.901680 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892062    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:38:13.901680 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892066    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:38:13.901680 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892070    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:38:13.901680 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892073    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:38:13.901680 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892076    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:38:13.901680 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892079    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:38:13.901680 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892082    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:38:13.901680 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892084    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:38:13.901680 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892087    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:38:13.901680 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892090    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:38:13.901680 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892093    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:38:13.901680 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892095    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:38:13.901680 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892098    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:38:13.901680 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892100    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:38:13.901680 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892103    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:38:13.902277 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892105    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:38:13.902277 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892108    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:38:13.902277 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892110    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:38:13.902277 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892113    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:38:13.902277 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892115    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:38:13.902277 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892118    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:38:13.902277 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892120    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:38:13.902277 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892123    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:38:13.902277 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892125    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:38:13.902277 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892129    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:38:13.902277 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892132    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:38:13.902277 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892134    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:38:13.902277 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892137    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:38:13.902277 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892139    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:38:13.902277 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892142    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:38:13.902277 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892144    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:38:13.902277 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892147    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:38:13.902277 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892149    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:38:13.902277 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892153    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:38:13.902759 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892156    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:38:13.902759 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892158    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:38:13.902759 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892161    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:38:13.902759 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892164    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:38:13.902759 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892166    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:38:13.902759 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892169    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:38:13.902759 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892171    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:38:13.902759 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892174    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:38:13.902759 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892177    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:38:13.902759 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892179    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:38:13.902759 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892182    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:38:13.902759 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892184    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:38:13.902759 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892187    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:38:13.902759 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892190    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:38:13.902759 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892192    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:38:13.902759 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892195    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:38:13.902759 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892197    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:38:13.902759 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892200    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:38:13.902759 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892202    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:38:13.903242 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892205    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:38:13.903242 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892207    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:38:13.903242 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892210    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:38:13.903242 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.892214    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:38:13.903242 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.892939    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 12:38:13.903242 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.901073    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 12:38:13.903242 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.901091    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 12:38:13.903242 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.901110    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 12:38:13.903242 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.901116    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 12:38:13.903242 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901177    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:38:13.903242 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901182    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:38:13.903242 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901185    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:38:13.903242 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901188    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:38:13.903242 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901192    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:38:13.903242 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901194    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:38:13.903623 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901197    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:38:13.903623 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901200    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:38:13.903623 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901203    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:38:13.903623 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901205    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:38:13.903623 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901208    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:38:13.903623 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901211    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:38:13.903623 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901213    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:38:13.903623 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901216    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:38:13.903623 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901218    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:38:13.903623 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901221    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:38:13.903623 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901223    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:38:13.903623 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901226    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:38:13.903623 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901228    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:38:13.903623 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901231    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:38:13.903623 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901234    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:38:13.903623 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901236    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:38:13.903623 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901238    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:38:13.903623 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901241    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:38:13.903623 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901245    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:38:13.903623 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901249    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:38:13.904130 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901252    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:38:13.904130 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901255    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:38:13.904130 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901258    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:38:13.904130 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901267    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:38:13.904130 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901269    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:38:13.904130 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901272    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:38:13.904130 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901275    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:38:13.904130 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901277    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:38:13.904130 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901279    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:38:13.904130 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901282    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:38:13.904130 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901284    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:38:13.904130 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901287    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:38:13.904130 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901289    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:38:13.904130 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901292    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:38:13.904130 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901294    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:38:13.904130 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901297    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:38:13.904130 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901299    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:38:13.904130 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901301    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:38:13.904130 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901304    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:38:13.904130 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901306    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:38:13.904618 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901309    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:38:13.904618 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901312    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:38:13.904618 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901314    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:38:13.904618 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901316    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:38:13.904618 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901319    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:38:13.904618 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901321    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:38:13.904618 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901323    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:38:13.904618 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901326    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:38:13.904618 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901328    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:38:13.904618 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901331    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:38:13.904618 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901334    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:38:13.904618 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901336    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:38:13.904618 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901338    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:38:13.904618 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901341    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:38:13.904618 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901344    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:38:13.904618 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901347    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:38:13.904618 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901358    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:38:13.904618 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901360    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:38:13.904618 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901363    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:38:13.904618 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901365    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:38:13.905197 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901368    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:38:13.905197 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901370    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:38:13.905197 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901373    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:38:13.905197 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901377    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:38:13.905197 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901380    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:38:13.905197 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901383    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:38:13.905197 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901385    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:38:13.905197 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901388    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:38:13.905197 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901390    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:38:13.905197 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901393    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:38:13.905197 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901396    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:38:13.905197 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901399    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:38:13.905197 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901401    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:38:13.905197 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901404    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:38:13.905197 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901406    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:38:13.905197 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901409    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:38:13.905197 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901411    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:38:13.905197 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901413    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:38:13.905197 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901416    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:38:13.905687 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901424    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:38:13.905687 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901427    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:38:13.905687 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901429    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:38:13.905687 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901432    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:38:13.905687 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901435    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:38:13.905687 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901437    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:38:13.905687 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901441    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:38:13.905687 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901445    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:38:13.905687 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901448    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:38:13.905687 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901451    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:38:13.905687 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901460    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:38:13.905687 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901463    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:38:13.905687 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901466    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:38:13.905687 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901468    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:38:13.906036 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.901473    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 12:38:13.906036 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901614    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:38:13.906036 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901619    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:38:13.906036 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901622    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:38:13.906036 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901625    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:38:13.906036 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901627    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:38:13.906036 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901630    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:38:13.906036 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901633    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:38:13.906036 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901636    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:38:13.906036 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901638    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:38:13.906036 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901641    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:38:13.906036 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901643    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:38:13.906036 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901646    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:38:13.906036 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901648    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:38:13.906036 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901651    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:38:13.906413 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901653    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:38:13.906413 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901656    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:38:13.906413 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901658    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:38:13.906413 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901660    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:38:13.906413 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901663    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:38:13.906413 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901665    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:38:13.906413 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901668    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:38:13.906413 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901670    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:38:13.906413 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901672    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:38:13.906413 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901675    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:38:13.906413 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901678    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:38:13.906413 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901680    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:38:13.906413 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901683    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:38:13.906413 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901685    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:38:13.906413 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901696    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:38:13.906413 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901700    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:38:13.906413 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901703    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:38:13.906413 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901706    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:38:13.906413 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901709    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:38:13.906880 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901711    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:38:13.906880 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901714    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:38:13.906880 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901717    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:38:13.906880 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901719    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:38:13.906880 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901722    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:38:13.906880 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901725    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:38:13.906880 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901728    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:38:13.906880 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901730    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:38:13.906880 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901733    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:38:13.906880 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901735    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:38:13.906880 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901737    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:38:13.906880 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901740    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:38:13.906880 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901742    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:38:13.906880 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901745    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:38:13.906880 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901747    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:38:13.906880 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901750    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:38:13.906880 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901752    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:38:13.906880 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901755    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:38:13.906880 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901757    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:38:13.906880 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901761    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:38:13.907371 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901764    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:38:13.907371 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901767    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:38:13.907371 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901769    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:38:13.907371 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901773    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:38:13.907371 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901775    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:38:13.907371 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901778    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:38:13.907371 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901780    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:38:13.907371 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901783    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:38:13.907371 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901790    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:38:13.907371 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901793    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:38:13.907371 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901795    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:38:13.907371 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901798    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:38:13.907371 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901800    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:38:13.907371 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901803    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:38:13.907371 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901805    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:38:13.907371 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901808    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:38:13.907371 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901810    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:38:13.907371 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901812    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:38:13.907371 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901815    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:38:13.907371 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901817    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:38:13.907860 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901820    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:38:13.907860 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901822    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:38:13.907860 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901824    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:38:13.907860 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901827    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:38:13.907860 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901829    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:38:13.907860 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901832    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:38:13.907860 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901834    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:38:13.907860 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901837    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:38:13.907860 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901839    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:38:13.907860 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901842    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:38:13.907860 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901844    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:38:13.907860 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901846    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:38:13.907860 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901849    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:38:13.907860 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901851    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:38:13.907860 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901854    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:38:13.907860 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901856    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:38:13.907860 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901859    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:38:13.907860 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901862    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:38:13.907860 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901866    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:38:13.908340 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901869    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:38:13.908340 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901871    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:38:13.908340 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901880    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:38:13.908340 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901883    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:38:13.908340 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901885    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:38:13.908340 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901888    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:38:13.908340 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:13.901891    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:38:13.908340 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.901896    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 12:38:13.908340 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.902073    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 12:38:13.908340 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.904572    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 12:38:13.908340 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.905698    2577 server.go:1019] "Starting client certificate rotation"
Sep 16 12:38:13.908340 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.905808    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 12:38:13.908340 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.905853    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 12:38:13.908670 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.908594    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 12:38:13.915587 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.915565    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 12:38:13.925977 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.925962    2577 log.go:25] "Validated CRI v1 runtime API"
Sep 16 12:38:13.929635 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.929620    2577 log.go:25] "Validated CRI v1 image API"
Sep 16 12:38:13.930482 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.930466    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 12:38:13.936200 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.936179    2577 fs.go:135] Filesystem UUIDs: map[409e2fcd-bd4a-42c6-87e4-47b49b0e07d7:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b8cdb627-1959-4f2c-929a-3107d12049a1:/dev/nvme0n1p3]
Sep 16 12:38:13.936285 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.936198    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 12:38:13.940226 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.940206    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 12:38:13.941520 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.941413    2577 manager.go:216] Machine: {Timestamp:2026-09-16 12:38:13.939871025 +0000 UTC m=+0.387103658 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099674 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2188eb195aad1a327f7569b9d326a6 SystemUUID:ec2188eb-195a-ad1a-327f-7569b9d326a6 BootID:ae35a141-9d02-4c29-9561-462813303c48 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d7:83:7d:6f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d7:83:7d:6f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:be:b0:5f:cb:62 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 12:38:13.941520 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.941516    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 12:38:13.941675 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.941655    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 12:38:13.945138 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.945117    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 12:38:13.945263 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.945141    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-190.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 12:38:13.945978 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.945968    2577 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 12:38:13.946011 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.945982    2577 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 12:38:13.946011 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.945990    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 12:38:13.946846 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.946835    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 12:38:13.948153 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.948143    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 12:38:13.952957 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.952945    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 12:38:13.953104 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.953094    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 12:38:13.953145 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.953125    2577 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 12:38:13.953145 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.953139    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 12:38:13.953199 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.953157    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 12:38:13.953199 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.953168    2577 kubelet.go:398] "Adding apiserver pod source"
Sep 16 12:38:13.953199 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.953177    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 12:38:13.953529 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.953518    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 12:38:13.953575 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.953536    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 12:38:13.955899 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.955885    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 12:38:13.958101 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.958088    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 12:38:13.958150 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.958118    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 12:38:13.960229 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.960218    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 12:38:13.960275 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.960235    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 12:38:13.960275 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.960241    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 12:38:13.960275 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.960247    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 12:38:13.960275 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.960260    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 12:38:13.960275 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.960268    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 12:38:13.960406 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.960282    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 12:38:13.960406 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.960287    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 12:38:13.960406 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.960293    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 12:38:13.960406 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.960298    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 12:38:13.960406 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.960307    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 12:38:13.960406 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.960312    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 12:38:13.961233 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.961222    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 12:38:13.961233 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.961233    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 12:38:13.964924 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:13.964903    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-190.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 12:38:13.965049 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:13.965003    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 12:38:13.965100 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.965084    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-190.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 12:38:13.965100 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.965095    2577 server.go:1268] "Started kubelet"
Sep 16 12:38:13.965165 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.965132    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 12:38:13.965250 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.965172    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 12:38:13.965305 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.965269    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 12:38:13.970100 ip-10-0-133-190 systemd[1]: Started Kubernetes Kubelet.
Sep 16 12:38:13.971969 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.970068    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 12:38:13.971969 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.971399    2577 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 12:38:13.977866 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:13.976670    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-190.ec2.internal.18d5cd6cc3389a1e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-190.ec2.internal,UID:ip-10-0-133-190.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-190.ec2.internal,},FirstTimestamp:2026-09-16 12:38:13.96506883 +0000 UTC m=+0.412301463,LastTimestamp:2026-09-16 12:38:13.96506883 +0000 UTC m=+0.412301463,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-190.ec2.internal,}"
Sep 16 12:38:13.977964 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:13.977949    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 12:38:13.980207 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.980193    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 12:38:13.980207 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.980203    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 12:38:13.980316 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:13.980279    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-190.ec2.internal\" not found"
Sep 16 12:38:13.980978 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.980963    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 12:38:13.981133 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.981101    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 12:38:13.981222 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.981047    2577 factory.go:55] Registering systemd factory
Sep 16 12:38:13.981222 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.980991    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 12:38:13.981222 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.981198    2577 factory.go:223] Registration of the systemd container factory successfully
Sep 16 12:38:13.981377 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.981265    2577 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 12:38:13.981377 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.981271    2577 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 12:38:13.981532 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.981514    2577 factory.go:153] Registering CRI-O factory
Sep 16 12:38:13.981532 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.981532    2577 factory.go:223] Registration of the crio container factory successfully
Sep 16 12:38:13.981631 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.981584    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 12:38:13.981631 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.981604    2577 factory.go:103] Registering Raw factory
Sep 16 12:38:13.981631 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.981619    2577 manager.go:1196] Started watching for new ooms in manager
Sep 16 12:38:13.982021 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.982005    2577 manager.go:349] Starting recovery of all containers
Sep 16 12:38:13.984221 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:13.984197    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 12:38:13.984221 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:13.984184    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-190.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 12:38:13.991769 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.991752    2577 manager.go:354] Recovery completed
Sep 16 12:38:13.995083 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.995064    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nvd27"
Sep 16 12:38:13.996863 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.996770    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:38:13.998360 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.998344    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-190.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:38:13.998433 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.998374    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-190.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:38:13.998433 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.998387    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-190.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:38:13.998934 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.998921    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 12:38:13.998934 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.998932    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 12:38:13.999052 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.998936    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:38:13.999052 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.998946    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 12:38:13.999589 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.999576    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-190.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:38:13.999634 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.999602    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-190.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:38:13.999634 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:13.999612    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-190.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:38:14.000327 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:14.000255    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-190.ec2.internal.18d5cd6cc5349126  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-190.ec2.internal,UID:ip-10-0-133-190.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-190.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-190.ec2.internal,},FirstTimestamp:2026-09-16 12:38:13.998358822 +0000 UTC m=+0.445591454,LastTimestamp:2026-09-16 12:38:13.998358822 +0000 UTC m=+0.445591454,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-190.ec2.internal,}"
Sep 16 12:38:14.000935 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.000922    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nvd27"
Sep 16 12:38:14.002280 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.002267    2577 policy_none.go:49] "None policy: Start"
Sep 16 12:38:14.002322 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.002287    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 12:38:14.002322 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.002298    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 12:38:14.004199 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.004188    2577 policy_none.go:47] "Start"
Sep 16 12:38:14.035596 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.035581    2577 manager.go:337] "Starting Device Plugin manager"
Sep 16 12:38:14.058646 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:14.035617    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 12:38:14.058646 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.035628    2577 server.go:87] "Starting device plugin registration server"
Sep 16 12:38:14.058646 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.035891    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 12:38:14.058646 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.035903    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 12:38:14.058646 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.035993    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 12:38:14.058646 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.036087    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 12:38:14.058646 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.036096    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 12:38:14.058646 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:14.036611    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 12:38:14.058646 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:14.036664    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-190.ec2.internal\" not found"
Sep 16 12:38:14.058646 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.046557    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 12:38:14.058646 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.047841    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 12:38:14.058646 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.047893    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 12:38:14.058646 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.047916    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 12:38:14.058646 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:14.047967    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 12:38:14.058646 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.050591    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:38:14.137099 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.136970    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:38:14.137965 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.137949    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-190.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:38:14.138049 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.137979    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-190.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:38:14.138049 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.137989    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-190.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:38:14.138049 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.138011    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-190.ec2.internal"
Sep 16 12:38:14.149050 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.149007    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-190.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-190.ec2.internal"]
Sep 16 12:38:14.149104 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.149095    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:38:14.150634 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.150620    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-190.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:38:14.150700 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.150644    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-190.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:38:14.150700 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.150654    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-190.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:38:14.151611 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.151599    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-190.ec2.internal"
Sep 16 12:38:14.151647 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:14.151618    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-190.ec2.internal\": node \"ip-10-0-133-190.ec2.internal\" not found"
Sep 16 12:38:14.152910 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.152899    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:38:14.153068 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.153054    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-190.ec2.internal"
Sep 16 12:38:14.153123 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.153090    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:38:14.153649 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.153630    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-190.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:38:14.153753 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.153653    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-190.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:38:14.153753 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.153666    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-190.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:38:14.153753 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.153688    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-190.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:38:14.153753 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.153708    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-190.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:38:14.153753 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.153718    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-190.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:38:14.155917 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.155900    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-190.ec2.internal"
Sep 16 12:38:14.156007 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.155931    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:38:14.156637 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.156620    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-190.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:38:14.156709 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.156648    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-190.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:38:14.156709 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.156658    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-190.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:38:14.174627 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:14.174605    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-190.ec2.internal\" not found" node="ip-10-0-133-190.ec2.internal"
Sep 16 12:38:14.174725 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:14.174644    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-190.ec2.internal\" not found"
Sep 16 12:38:14.178519 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:14.178502    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-190.ec2.internal\" not found" node="ip-10-0-133-190.ec2.internal"
Sep 16 12:38:14.182240 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.182222    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/71c783a701963269184ce00c0e4aab2a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-190.ec2.internal\" (UID: \"71c783a701963269184ce00c0e4aab2a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-190.ec2.internal"
Sep 16 12:38:14.182320 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.182252    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71c783a701963269184ce00c0e4aab2a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-190.ec2.internal\" (UID: \"71c783a701963269184ce00c0e4aab2a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-190.ec2.internal"
Sep 16 12:38:14.182320 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.182280    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dad4cbabbaf3126a41405ddf99389dd4-config\") pod \"kube-apiserver-proxy-ip-10-0-133-190.ec2.internal\" (UID: \"dad4cbabbaf3126a41405ddf99389dd4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-190.ec2.internal"
Sep 16 12:38:14.274924 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:14.274899    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-190.ec2.internal\" not found"
Sep 16 12:38:14.283264 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.283248    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/71c783a701963269184ce00c0e4aab2a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-190.ec2.internal\" (UID: \"71c783a701963269184ce00c0e4aab2a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-190.ec2.internal"
Sep 16 12:38:14.283313 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.283275    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71c783a701963269184ce00c0e4aab2a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-190.ec2.internal\" (UID: \"71c783a701963269184ce00c0e4aab2a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-190.ec2.internal"
Sep 16 12:38:14.283313 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.283292    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dad4cbabbaf3126a41405ddf99389dd4-config\") pod \"kube-apiserver-proxy-ip-10-0-133-190.ec2.internal\" (UID: \"dad4cbabbaf3126a41405ddf99389dd4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-190.ec2.internal"
Sep 16 12:38:14.283409 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.283332    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dad4cbabbaf3126a41405ddf99389dd4-config\") pod \"kube-apiserver-proxy-ip-10-0-133-190.ec2.internal\" (UID: \"dad4cbabbaf3126a41405ddf99389dd4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-190.ec2.internal"
Sep 16 12:38:14.283409 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.283360    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/71c783a701963269184ce00c0e4aab2a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-190.ec2.internal\" (UID: \"71c783a701963269184ce00c0e4aab2a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-190.ec2.internal"
Sep 16 12:38:14.283409 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.283371    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71c783a701963269184ce00c0e4aab2a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-190.ec2.internal\" (UID: \"71c783a701963269184ce00c0e4aab2a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-190.ec2.internal"
Sep 16 12:38:14.375370 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:14.375340    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-190.ec2.internal\" not found"
Sep 16 12:38:14.476139 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:14.476058    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-190.ec2.internal\" not found"
Sep 16 12:38:14.478216 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.478198    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-190.ec2.internal"
Sep 16 12:38:14.482280 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.482264    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-190.ec2.internal"
Sep 16 12:38:14.576733 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:14.576695    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-190.ec2.internal\" not found"
Sep 16 12:38:14.677288 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:14.677257    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-190.ec2.internal\" not found"
Sep 16 12:38:14.777829 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:14.777745    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-190.ec2.internal\" not found"
Sep 16 12:38:14.878457 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:14.878298    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-190.ec2.internal\" not found"
Sep 16 12:38:14.887934 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.887915    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:38:14.905347 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.905324    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 12:38:14.905513 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.905473    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 12:38:14.905559 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:14.905473    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 12:38:14.979557 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:14.979511    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-190.ec2.internal\" not found"
Sep 16 12:38:15.006732 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.006700    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 12:33:14 +0000 UTC" deadline="2028-06-24 09:37:21.964168018 +0000 UTC"
Sep 16 12:38:15.006732 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.006730    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15524h59m6.957441916s"
Sep 16 12:38:15.030840 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.030793    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:38:15.065170 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:15.065127    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddad4cbabbaf3126a41405ddf99389dd4.slice/crio-7ecd30919025e715b3d5ddb60000603a80535f7ffeec53289649050b9466cd93 WatchSource:0}: Error finding container 7ecd30919025e715b3d5ddb60000603a80535f7ffeec53289649050b9466cd93: Status 404 returned error can't find the container with id 7ecd30919025e715b3d5ddb60000603a80535f7ffeec53289649050b9466cd93
Sep 16 12:38:15.065472 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:15.065449    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c783a701963269184ce00c0e4aab2a.slice/crio-518a8cef273ff3c1264dd4da0c81ca1633c7407f9a3163644147d3c6f8318f0f WatchSource:0}: Error finding container 518a8cef273ff3c1264dd4da0c81ca1633c7407f9a3163644147d3c6f8318f0f: Status 404 returned error can't find the container with id 518a8cef273ff3c1264dd4da0c81ca1633c7407f9a3163644147d3c6f8318f0f
Sep 16 12:38:15.069335 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.069321    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:38:15.079918 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:15.079900    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-190.ec2.internal\" not found"
Sep 16 12:38:15.154135 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.154107    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:38:15.180474 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.180453    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-190.ec2.internal"
Sep 16 12:38:15.195393 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.195374    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 12:38:15.196518 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.196505    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-190.ec2.internal"
Sep 16 12:38:15.207379 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.207362    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 12:38:15.766569 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.766535    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:38:15.954264 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.954157    2577 apiserver.go:52] "Watching apiserver"
Sep 16 12:38:15.966044 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.965999    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 12:38:15.966455 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.966433    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-8bxln","openshift-multus/multus-additional-cni-plugins-9f6jr","openshift-multus/multus-l6tn5","openshift-multus/network-metrics-daemon-tmfpk","openshift-network-diagnostics/network-check-target-xmbgf","openshift-network-operator/iptables-alerter-29bsx","kube-system/global-pull-secret-syncer-bwb7l","kube-system/kube-apiserver-proxy-ip-10-0-133-190.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6psq","openshift-dns/node-resolver-t47mb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-190.ec2.internal","openshift-ovn-kubernetes/ovnkube-node-6g82p","kube-system/konnectivity-agent-j2kjn","openshift-cluster-node-tuning-operator/tuned-ftpbx"]
Sep 16 12:38:15.969150 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.969129    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwb7l"
Sep 16 12:38:15.969274 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:15.969200    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwb7l" podUID="342f831e-0ffe-4582-b616-e0ce14364b32"
Sep 16 12:38:15.971524 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.971500    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j2kjn"
Sep 16 12:38:15.975590 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.974101    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:15.975590 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.975020    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4j7jj\""
Sep 16 12:38:15.975590 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.975238    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 12:38:15.975590 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.975449    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 12:38:15.976410 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.976389    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmfpk"
Sep 16 12:38:15.976499 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:15.976457    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmfpk" podUID="aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5"
Sep 16 12:38:15.979649 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.977792    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 12:38:15.979649 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.977815    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 12:38:15.979649 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.977825    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-q7cjz\""
Sep 16 12:38:15.979649 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.977823    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 12:38:15.979649 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.977823    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 12:38:15.979649 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.978794    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmbgf"
Sep 16 12:38:15.979649 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:15.978870    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmbgf" podUID="076f350f-8aef-4c0f-882a-7ac207cf1535"
Sep 16 12:38:15.981024 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.981004    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 12:38:15.984186 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.983314    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-29bsx"
Sep 16 12:38:15.984704 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.984684    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6psq"
Sep 16 12:38:15.986529 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.986509    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:38:15.986764 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.986746    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4jq2w\""
Sep 16 12:38:15.986844 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.986764    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 12:38:15.986896 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.986849    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 12:38:15.987238 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.987112    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8bxln"
Sep 16 12:38:15.987238 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.987187    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 12:38:15.987661 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.987645    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 12:38:15.988259 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.988242    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2scx4\""
Sep 16 12:38:15.988442 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.988425    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 12:38:15.990226 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.989760    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9f6jr"
Sep 16 12:38:15.990992 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.990974    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 12:38:15.991484 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.991311    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 12:38:15.991579 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.991549    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 12:38:15.991812 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.991764    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-29zxv\""
Sep 16 12:38:15.992311 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.992286    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t47mb"
Sep 16 12:38:15.992762 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.992740    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 12:38:15.992895 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.992875    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 12:38:15.992947 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.992901    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5-metrics-certs\") pod \"network-metrics-daemon-tmfpk\" (UID: \"aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5\") " pod="openshift-multus/network-metrics-daemon-tmfpk"
Sep 16 12:38:15.992947 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.992927    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9a4ba4d0-a20b-4ed7-9fc7-92bf3943faf6-host-slash\") pod \"iptables-alerter-29bsx\" (UID: \"9a4ba4d0-a20b-4ed7-9fc7-92bf3943faf6\") " pod="openshift-network-operator/iptables-alerter-29bsx"
Sep 16 12:38:15.993053 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.992958    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5697a20-a56e-4bc9-94c6-73711e6c52ba-cnibin\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:15.993053 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.992981    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5697a20-a56e-4bc9-94c6-73711e6c52ba-host-run-netns\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:15.993053 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.993006    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e5697a20-a56e-4bc9-94c6-73711e6c52ba-hostroot\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:15.993053 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.993049    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jnqk\" (UniqueName: \"kubernetes.io/projected/e5697a20-a56e-4bc9-94c6-73711e6c52ba-kube-api-access-9jnqk\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:15.993292 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.993073    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5697a20-a56e-4bc9-94c6-73711e6c52ba-host-var-lib-cni-bin\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:15.993292 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.993105    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c9fcc2dc-c3c3-401c-8534-95cd3908332e-agent-certs\") pod \"konnectivity-agent-j2kjn\" (UID: \"c9fcc2dc-c3c3-401c-8534-95cd3908332e\") " pod="kube-system/konnectivity-agent-j2kjn"
Sep 16 12:38:15.993292 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.993123    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e5697a20-a56e-4bc9-94c6-73711e6c52ba-host-var-lib-cni-multus\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:15.993292 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.993149    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e5697a20-a56e-4bc9-94c6-73711e6c52ba-multus-daemon-config\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:15.993292 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.993175    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/342f831e-0ffe-4582-b616-e0ce14364b32-kubelet-config\") pod \"global-pull-secret-syncer-bwb7l\" (UID: \"342f831e-0ffe-4582-b616-e0ce14364b32\") " pod="kube-system/global-pull-secret-syncer-bwb7l"
Sep 16 12:38:15.993292 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.993242    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c9fcc2dc-c3c3-401c-8534-95cd3908332e-konnectivity-ca\") pod \"konnectivity-agent-j2kjn\" (UID: \"c9fcc2dc-c3c3-401c-8534-95cd3908332e\") " pod="kube-system/konnectivity-agent-j2kjn"
Sep 16 12:38:15.993593 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.993298    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e5697a20-a56e-4bc9-94c6-73711e6c52ba-cni-binary-copy\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:15.993593 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.993336    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4bz7g\""
Sep 16 12:38:15.993593 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.993347    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e5697a20-a56e-4bc9-94c6-73711e6c52ba-host-run-k8s-cni-cncf-io\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:15.993593 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.993402    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e5697a20-a56e-4bc9-94c6-73711e6c52ba-host-run-multus-certs\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:15.993593 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.993428    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e5697a20-a56e-4bc9-94c6-73711e6c52ba-multus-socket-dir-parent\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:15.993593 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.993456    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pst9f\" (UniqueName: \"kubernetes.io/projected/aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5-kube-api-access-pst9f\") pod \"network-metrics-daemon-tmfpk\" (UID: \"aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5\") " pod="openshift-multus/network-metrics-daemon-tmfpk"
Sep 16 12:38:15.993593 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.993484    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/342f831e-0ffe-4582-b616-e0ce14364b32-dbus\") pod \"global-pull-secret-syncer-bwb7l\" (UID: \"342f831e-0ffe-4582-b616-e0ce14364b32\") " pod="kube-system/global-pull-secret-syncer-bwb7l"
Sep 16 12:38:15.993593 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.993510    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/342f831e-0ffe-4582-b616-e0ce14364b32-original-pull-secret\") pod \"global-pull-secret-syncer-bwb7l\" (UID: \"342f831e-0ffe-4582-b616-e0ce14364b32\") " pod="kube-system/global-pull-secret-syncer-bwb7l"
Sep 16 12:38:15.993593 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.993560    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5697a20-a56e-4bc9-94c6-73711e6c52ba-multus-cni-dir\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:15.993593 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.993584    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5697a20-a56e-4bc9-94c6-73711e6c52ba-multus-conf-dir\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:15.994000 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.993607    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5697a20-a56e-4bc9-94c6-73711e6c52ba-etc-kubernetes\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:15.994000 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.993634    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b58gn\" (UniqueName: \"kubernetes.io/projected/9a4ba4d0-a20b-4ed7-9fc7-92bf3943faf6-kube-api-access-b58gn\") pod \"iptables-alerter-29bsx\" (UID: \"9a4ba4d0-a20b-4ed7-9fc7-92bf3943faf6\") " pod="openshift-network-operator/iptables-alerter-29bsx"
Sep 16 12:38:15.994000 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.993677    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5697a20-a56e-4bc9-94c6-73711e6c52ba-host-var-lib-kubelet\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:15.994000 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.993705    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48bsc\" (UniqueName: \"kubernetes.io/projected/076f350f-8aef-4c0f-882a-7ac207cf1535-kube-api-access-48bsc\") pod \"network-check-target-xmbgf\" (UID: \"076f350f-8aef-4c0f-882a-7ac207cf1535\") " pod="openshift-network-diagnostics/network-check-target-xmbgf"
Sep 16 12:38:15.994000 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.993758    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9a4ba4d0-a20b-4ed7-9fc7-92bf3943faf6-iptables-alerter-script\") pod \"iptables-alerter-29bsx\" (UID: \"9a4ba4d0-a20b-4ed7-9fc7-92bf3943faf6\") " pod="openshift-network-operator/iptables-alerter-29bsx"
Sep 16 12:38:15.994000 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.993808    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5697a20-a56e-4bc9-94c6-73711e6c52ba-system-cni-dir\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:15.994767 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.994747    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:15.997808 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.997780    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 12:38:15.997925 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.997906    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 12:38:15.998401 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.998230    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-sllw2\""
Sep 16 12:38:15.999511 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.999497    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 12:38:15.999745 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.999720    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:38:15.999931 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:15.999917    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.000187 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.000088    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-f5pl2\""
Sep 16 12:38:16.000588 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.000428    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 12:38:16.003386 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.003369    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 12:38:16.003634 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.003618    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 12:38:16.003814 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.003798    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 12:38:16.004053 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.004036    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 12:38:16.005082 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.005061    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 12:38:16.005686 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.005473    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-k86ff\""
Sep 16 12:38:16.005686 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.005607    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 12:38:16.033493 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.033469    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-b29j7"
Sep 16 12:38:16.044231 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.043791    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-b29j7"
Sep 16 12:38:16.056000 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.055946    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-190.ec2.internal" event={"ID":"dad4cbabbaf3126a41405ddf99389dd4","Type":"ContainerStarted","Data":"7ecd30919025e715b3d5ddb60000603a80535f7ffeec53289649050b9466cd93"}
Sep 16 12:38:16.057614 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.057558    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-190.ec2.internal" event={"ID":"71c783a701963269184ce00c0e4aab2a","Type":"ContainerStarted","Data":"518a8cef273ff3c1264dd4da0c81ca1633c7407f9a3163644147d3c6f8318f0f"}
Sep 16 12:38:16.082697 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.082665    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 12:38:16.095010 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.094965    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-env-overrides\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.095168 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.095027    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a8b1fe11-76c6-4a4a-91a3-865667440c00-etc-selinux\") pod \"aws-ebs-csi-driver-node-s6psq\" (UID: \"a8b1fe11-76c6-4a4a-91a3-865667440c00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6psq"
Sep 16 12:38:16.095168 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.095071    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48bsc\" (UniqueName: \"kubernetes.io/projected/076f350f-8aef-4c0f-882a-7ac207cf1535-kube-api-access-48bsc\") pod \"network-check-target-xmbgf\" (UID: \"076f350f-8aef-4c0f-882a-7ac207cf1535\") " pod="openshift-network-diagnostics/network-check-target-xmbgf"
Sep 16 12:38:16.095168 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.095108    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw7ln\" (UniqueName: \"kubernetes.io/projected/a7e49775-cab4-4ccf-a510-9ed41e681e07-kube-api-access-pw7ln\") pod \"multus-additional-cni-plugins-9f6jr\" (UID: \"a7e49775-cab4-4ccf-a510-9ed41e681e07\") " pod="openshift-multus/multus-additional-cni-plugins-9f6jr"
Sep 16 12:38:16.095344 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.095204    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bl6x\" (UniqueName: \"kubernetes.io/projected/d3f21250-f182-428a-a11e-99f2a158ca61-kube-api-access-9bl6x\") pod \"node-ca-8bxln\" (UID: \"d3f21250-f182-428a-a11e-99f2a158ca61\") " pod="openshift-image-registry/node-ca-8bxln"
Sep 16 12:38:16.095344 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.095244    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a8b1fe11-76c6-4a4a-91a3-865667440c00-socket-dir\") pod \"aws-ebs-csi-driver-node-s6psq\" (UID: \"a8b1fe11-76c6-4a4a-91a3-865667440c00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6psq"
Sep 16 12:38:16.095448 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.095346    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5697a20-a56e-4bc9-94c6-73711e6c52ba-system-cni-dir\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:16.095448 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.095403    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9a4ba4d0-a20b-4ed7-9fc7-92bf3943faf6-host-slash\") pod \"iptables-alerter-29bsx\" (UID: \"9a4ba4d0-a20b-4ed7-9fc7-92bf3943faf6\") " pod="openshift-network-operator/iptables-alerter-29bsx"
Sep 16 12:38:16.095448 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.095442    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a7e49775-cab4-4ccf-a510-9ed41e681e07-system-cni-dir\") pod \"multus-additional-cni-plugins-9f6jr\" (UID: \"a7e49775-cab4-4ccf-a510-9ed41e681e07\") " pod="openshift-multus/multus-additional-cni-plugins-9f6jr"
Sep 16 12:38:16.095606 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.095501    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-ovn-node-metrics-cert\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.095606 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.095532    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d62f767-0859-4bff-a343-9dc4aac3c1f2-tmp\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.095606 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.095560    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e5697a20-a56e-4bc9-94c6-73711e6c52ba-hostroot\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:16.095606 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.095592    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jnqk\" (UniqueName: \"kubernetes.io/projected/e5697a20-a56e-4bc9-94c6-73711e6c52ba-kube-api-access-9jnqk\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:16.095810 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.095644    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-log-socket\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.095810 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.095654    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e5697a20-a56e-4bc9-94c6-73711e6c52ba-hostroot\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:16.095810 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.095671    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a7e49775-cab4-4ccf-a510-9ed41e681e07-cni-binary-copy\") pod \"multus-additional-cni-plugins-9f6jr\" (UID: \"a7e49775-cab4-4ccf-a510-9ed41e681e07\") " pod="openshift-multus/multus-additional-cni-plugins-9f6jr"
Sep 16 12:38:16.095810 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.095748    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c44032b0-4e54-42d9-945f-7d401e20d50c-hosts-file\") pod \"node-resolver-t47mb\" (UID: \"c44032b0-4e54-42d9-945f-7d401e20d50c\") " pod="openshift-dns/node-resolver-t47mb"
Sep 16 12:38:16.095810 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.095786    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d62f767-0859-4bff-a343-9dc4aac3c1f2-lib-modules\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.096065 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.095815    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5697a20-a56e-4bc9-94c6-73711e6c52ba-system-cni-dir\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:16.096065 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.095844    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e5697a20-a56e-4bc9-94c6-73711e6c52ba-multus-daemon-config\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:16.096065 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.095862    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a7e49775-cab4-4ccf-a510-9ed41e681e07-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9f6jr\" (UID: \"a7e49775-cab4-4ccf-a510-9ed41e681e07\") " pod="openshift-multus/multus-additional-cni-plugins-9f6jr"
Sep 16 12:38:16.096065 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.095865    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9a4ba4d0-a20b-4ed7-9fc7-92bf3943faf6-host-slash\") pod \"iptables-alerter-29bsx\" (UID: \"9a4ba4d0-a20b-4ed7-9fc7-92bf3943faf6\") " pod="openshift-network-operator/iptables-alerter-29bsx"
Sep 16 12:38:16.096065 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.095878    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-host-kubelet\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.096065 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.095903    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-ovnkube-script-lib\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.096065 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.095937    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcb9h\" (UniqueName: \"kubernetes.io/projected/5d62f767-0859-4bff-a343-9dc4aac3c1f2-kube-api-access-jcb9h\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.096065 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.095968    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c9fcc2dc-c3c3-401c-8534-95cd3908332e-konnectivity-ca\") pod \"konnectivity-agent-j2kjn\" (UID: \"c9fcc2dc-c3c3-401c-8534-95cd3908332e\") " pod="kube-system/konnectivity-agent-j2kjn"
Sep 16 12:38:16.096065 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.096027    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e5697a20-a56e-4bc9-94c6-73711e6c52ba-host-run-k8s-cni-cncf-io\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:16.096065 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.096043    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e5697a20-a56e-4bc9-94c6-73711e6c52ba-host-run-multus-certs\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:16.096065 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.096069    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-run-systemd\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.096611 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.096091    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e5697a20-a56e-4bc9-94c6-73711e6c52ba-multus-socket-dir-parent\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:16.096611 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.096131    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-host-slash\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.096611 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.096146    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3f21250-f182-428a-a11e-99f2a158ca61-host\") pod \"node-ca-8bxln\" (UID: \"d3f21250-f182-428a-a11e-99f2a158ca61\") " pod="openshift-image-registry/node-ca-8bxln"
Sep 16 12:38:16.096611 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.096170    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/342f831e-0ffe-4582-b616-e0ce14364b32-original-pull-secret\") pod \"global-pull-secret-syncer-bwb7l\" (UID: \"342f831e-0ffe-4582-b616-e0ce14364b32\") " pod="kube-system/global-pull-secret-syncer-bwb7l"
Sep 16 12:38:16.096611 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.096187    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5697a20-a56e-4bc9-94c6-73711e6c52ba-multus-conf-dir\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:16.096611 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.096215    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5697a20-a56e-4bc9-94c6-73711e6c52ba-etc-kubernetes\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:16.096611 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.096237    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-host-cni-netd\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.096611 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.096261    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d3f21250-f182-428a-a11e-99f2a158ca61-serviceca\") pod \"node-ca-8bxln\" (UID: \"d3f21250-f182-428a-a11e-99f2a158ca61\") " pod="openshift-image-registry/node-ca-8bxln"
Sep 16 12:38:16.096611 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.096373    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d62f767-0859-4bff-a343-9dc4aac3c1f2-var-lib-kubelet\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.096611 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.096421    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5697a20-a56e-4bc9-94c6-73711e6c52ba-host-var-lib-kubelet\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:16.096611 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.096456    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9a4ba4d0-a20b-4ed7-9fc7-92bf3943faf6-iptables-alerter-script\") pod \"iptables-alerter-29bsx\" (UID: \"9a4ba4d0-a20b-4ed7-9fc7-92bf3943faf6\") " pod="openshift-network-operator/iptables-alerter-29bsx"
Sep 16 12:38:16.096611 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.096484    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a7e49775-cab4-4ccf-a510-9ed41e681e07-cnibin\") pod \"multus-additional-cni-plugins-9f6jr\" (UID: \"a7e49775-cab4-4ccf-a510-9ed41e681e07\") " pod="openshift-multus/multus-additional-cni-plugins-9f6jr"
Sep 16 12:38:16.096611 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.096517    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-host-run-netns\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.096611 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.096563    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-etc-openvswitch\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.096611 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.096598    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.097132 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.096630    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a8b1fe11-76c6-4a4a-91a3-865667440c00-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s6psq\" (UID: \"a8b1fe11-76c6-4a4a-91a3-865667440c00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6psq"
Sep 16 12:38:16.097132 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.096671    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a8b1fe11-76c6-4a4a-91a3-865667440c00-sys-fs\") pod \"aws-ebs-csi-driver-node-s6psq\" (UID: \"a8b1fe11-76c6-4a4a-91a3-865667440c00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6psq"
Sep 16 12:38:16.097132 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.096692    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e5697a20-a56e-4bc9-94c6-73711e6c52ba-host-run-k8s-cni-cncf-io\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:16.097132 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.096704    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5-metrics-certs\") pod \"network-metrics-daemon-tmfpk\" (UID: \"aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5\") " pod="openshift-multus/network-metrics-daemon-tmfpk"
Sep 16 12:38:16.097132 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.096750    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-systemd-units\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.097132 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:16.096812    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:38:16.097132 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.096815    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-host-run-ovn-kubernetes\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.097132 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.096845    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-ovnkube-config\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.097132 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:16.096909    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5-metrics-certs podName:aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5 nodeName:}" failed. No retries permitted until 2026-09-16 12:38:16.596857391 +0000 UTC m=+3.044090061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5-metrics-certs") pod "network-metrics-daemon-tmfpk" (UID: "aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:38:16.097132 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.096939    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e5697a20-a56e-4bc9-94c6-73711e6c52ba-host-run-multus-certs\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:16.097132 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.097083    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e5697a20-a56e-4bc9-94c6-73711e6c52ba-multus-daemon-config\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:16.097132 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.097118    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e5697a20-a56e-4bc9-94c6-73711e6c52ba-multus-socket-dir-parent\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:16.097681 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.096946    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5d62f767-0859-4bff-a343-9dc4aac3c1f2-etc-modprobe-d\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.097681 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.097577    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c9fcc2dc-c3c3-401c-8534-95cd3908332e-konnectivity-ca\") pod \"konnectivity-agent-j2kjn\" (UID: \"c9fcc2dc-c3c3-401c-8534-95cd3908332e\") " pod="kube-system/konnectivity-agent-j2kjn"
Sep 16 12:38:16.098009 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.097990    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9a4ba4d0-a20b-4ed7-9fc7-92bf3943faf6-iptables-alerter-script\") pod \"iptables-alerter-29bsx\" (UID: \"9a4ba4d0-a20b-4ed7-9fc7-92bf3943faf6\") " pod="openshift-network-operator/iptables-alerter-29bsx"
Sep 16 12:38:16.098094 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:16.098059    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:38:16.098094 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.098086    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5697a20-a56e-4bc9-94c6-73711e6c52ba-multus-conf-dir\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:16.098177 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.098126    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5697a20-a56e-4bc9-94c6-73711e6c52ba-etc-kubernetes\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:16.098177 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.098157    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5d62f767-0859-4bff-a343-9dc4aac3c1f2-etc-sysconfig\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.098263 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.098202    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5697a20-a56e-4bc9-94c6-73711e6c52ba-host-var-lib-kubelet\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:16.098297 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:16.098268    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/342f831e-0ffe-4582-b616-e0ce14364b32-original-pull-secret podName:342f831e-0ffe-4582-b616-e0ce14364b32 nodeName:}" failed. No retries permitted until 2026-09-16 12:38:16.598255792 +0000 UTC m=+3.045488428 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/342f831e-0ffe-4582-b616-e0ce14364b32-original-pull-secret") pod "global-pull-secret-syncer-bwb7l" (UID: "342f831e-0ffe-4582-b616-e0ce14364b32") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:38:16.098345 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.098302    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d62f767-0859-4bff-a343-9dc4aac3c1f2-etc-kubernetes\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.098393 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.098365    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5697a20-a56e-4bc9-94c6-73711e6c52ba-cnibin\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:16.098441 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.098406    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5697a20-a56e-4bc9-94c6-73711e6c52ba-host-run-netns\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:16.098486 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.098443    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqfdl\" (UniqueName: \"kubernetes.io/projected/c44032b0-4e54-42d9-945f-7d401e20d50c-kube-api-access-dqfdl\") pod \"node-resolver-t47mb\" (UID: \"c44032b0-4e54-42d9-945f-7d401e20d50c\") " pod="openshift-dns/node-resolver-t47mb"
Sep 16 12:38:16.098538 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.098473    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjq9p\" (UniqueName: \"kubernetes.io/projected/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-kube-api-access-cjq9p\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.098538 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.098495    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5697a20-a56e-4bc9-94c6-73711e6c52ba-host-run-netns\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:16.098538 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.098525    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5d62f767-0859-4bff-a343-9dc4aac3c1f2-etc-sysctl-conf\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.098702 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.098553    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5d62f767-0859-4bff-a343-9dc4aac3c1f2-run\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.098702 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.098553    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5697a20-a56e-4bc9-94c6-73711e6c52ba-cnibin\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:16.098702 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.098613    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d62f767-0859-4bff-a343-9dc4aac3c1f2-host\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.098702 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.098656    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5d62f767-0859-4bff-a343-9dc4aac3c1f2-etc-tuned\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.098702 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.098696    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5697a20-a56e-4bc9-94c6-73711e6c52ba-host-var-lib-cni-bin\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:16.099015 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.098726    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-var-lib-openvswitch\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.099015 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.098752    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d62f767-0859-4bff-a343-9dc4aac3c1f2-sys\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.099015 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.098818    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5697a20-a56e-4bc9-94c6-73711e6c52ba-host-var-lib-cni-bin\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:16.099015 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.098826    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c9fcc2dc-c3c3-401c-8534-95cd3908332e-agent-certs\") pod \"konnectivity-agent-j2kjn\" (UID: \"c9fcc2dc-c3c3-401c-8534-95cd3908332e\") " pod="kube-system/konnectivity-agent-j2kjn"
Sep 16 12:38:16.099015 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.098861    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e5697a20-a56e-4bc9-94c6-73711e6c52ba-host-var-lib-cni-multus\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:16.099015 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.098932    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-node-log\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.099015 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.098958    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e5697a20-a56e-4bc9-94c6-73711e6c52ba-host-var-lib-cni-multus\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:16.099015 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.099002    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a8b1fe11-76c6-4a4a-91a3-865667440c00-device-dir\") pod \"aws-ebs-csi-driver-node-s6psq\" (UID: \"a8b1fe11-76c6-4a4a-91a3-865667440c00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6psq"
Sep 16 12:38:16.099438 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.099058    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpmt6\" (UniqueName: \"kubernetes.io/projected/a8b1fe11-76c6-4a4a-91a3-865667440c00-kube-api-access-jpmt6\") pod \"aws-ebs-csi-driver-node-s6psq\" (UID: \"a8b1fe11-76c6-4a4a-91a3-865667440c00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6psq"
Sep 16 12:38:16.099438 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.099093    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/342f831e-0ffe-4582-b616-e0ce14364b32-kubelet-config\") pod \"global-pull-secret-syncer-bwb7l\" (UID: \"342f831e-0ffe-4582-b616-e0ce14364b32\") " pod="kube-system/global-pull-secret-syncer-bwb7l"
Sep 16 12:38:16.099438 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.099119    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e5697a20-a56e-4bc9-94c6-73711e6c52ba-cni-binary-copy\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:16.099438 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.099161    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-run-openvswitch\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.099438 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.099213    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pst9f\" (UniqueName: \"kubernetes.io/projected/aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5-kube-api-access-pst9f\") pod \"network-metrics-daemon-tmfpk\" (UID: \"aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5\") " pod="openshift-multus/network-metrics-daemon-tmfpk"
Sep 16 12:38:16.099438 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.099229    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/342f831e-0ffe-4582-b616-e0ce14364b32-kubelet-config\") pod \"global-pull-secret-syncer-bwb7l\" (UID: \"342f831e-0ffe-4582-b616-e0ce14364b32\") " pod="kube-system/global-pull-secret-syncer-bwb7l"
Sep 16 12:38:16.099438 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.099259    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a7e49775-cab4-4ccf-a510-9ed41e681e07-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9f6jr\" (UID: \"a7e49775-cab4-4ccf-a510-9ed41e681e07\") " pod="openshift-multus/multus-additional-cni-plugins-9f6jr"
Sep 16 12:38:16.099438 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.099262    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 12:38:16.099438 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.099302    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a7e49775-cab4-4ccf-a510-9ed41e681e07-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9f6jr\" (UID: \"a7e49775-cab4-4ccf-a510-9ed41e681e07\") " pod="openshift-multus/multus-additional-cni-plugins-9f6jr"
Sep 16 12:38:16.099438 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.099329    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c44032b0-4e54-42d9-945f-7d401e20d50c-tmp-dir\") pod \"node-resolver-t47mb\" (UID: \"c44032b0-4e54-42d9-945f-7d401e20d50c\") " pod="openshift-dns/node-resolver-t47mb"
Sep 16 12:38:16.099438 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.099397    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-host-cni-bin\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.099438 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.099428    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5d62f767-0859-4bff-a343-9dc4aac3c1f2-etc-sysctl-d\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.099907 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.099457    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5d62f767-0859-4bff-a343-9dc4aac3c1f2-etc-systemd\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.099907 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.099481    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a8b1fe11-76c6-4a4a-91a3-865667440c00-registration-dir\") pod \"aws-ebs-csi-driver-node-s6psq\" (UID: \"a8b1fe11-76c6-4a4a-91a3-865667440c00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6psq"
Sep 16 12:38:16.099907 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.099515    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/342f831e-0ffe-4582-b616-e0ce14364b32-dbus\") pod \"global-pull-secret-syncer-bwb7l\" (UID: \"342f831e-0ffe-4582-b616-e0ce14364b32\") " pod="kube-system/global-pull-secret-syncer-bwb7l"
Sep 16 12:38:16.099907 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.099541    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5697a20-a56e-4bc9-94c6-73711e6c52ba-multus-cni-dir\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:16.099907 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.099570    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b58gn\" (UniqueName: \"kubernetes.io/projected/9a4ba4d0-a20b-4ed7-9fc7-92bf3943faf6-kube-api-access-b58gn\") pod \"iptables-alerter-29bsx\" (UID: \"9a4ba4d0-a20b-4ed7-9fc7-92bf3943faf6\") " pod="openshift-network-operator/iptables-alerter-29bsx"
Sep 16 12:38:16.099907 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.099650    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e5697a20-a56e-4bc9-94c6-73711e6c52ba-cni-binary-copy\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:16.099907 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.099682    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-run-ovn\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.099907 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.099706    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5697a20-a56e-4bc9-94c6-73711e6c52ba-multus-cni-dir\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:16.099907 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.099760    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/342f831e-0ffe-4582-b616-e0ce14364b32-dbus\") pod \"global-pull-secret-syncer-bwb7l\" (UID: \"342f831e-0ffe-4582-b616-e0ce14364b32\") " pod="kube-system/global-pull-secret-syncer-bwb7l"
Sep 16 12:38:16.104166 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.104093    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c9fcc2dc-c3c3-401c-8534-95cd3908332e-agent-certs\") pod \"konnectivity-agent-j2kjn\" (UID: \"c9fcc2dc-c3c3-401c-8534-95cd3908332e\") " pod="kube-system/konnectivity-agent-j2kjn"
Sep 16 12:38:16.106173 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:16.106152    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:38:16.106294 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:16.106179    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:38:16.106294 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:16.106192    2577 projected.go:196] Error preparing data for projected volume kube-api-access-48bsc for pod openshift-network-diagnostics/network-check-target-xmbgf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:38:16.106294 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:16.106249    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/076f350f-8aef-4c0f-882a-7ac207cf1535-kube-api-access-48bsc podName:076f350f-8aef-4c0f-882a-7ac207cf1535 nodeName:}" failed. No retries permitted until 2026-09-16 12:38:16.606233262 +0000 UTC m=+3.053465900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-48bsc" (UniqueName: "kubernetes.io/projected/076f350f-8aef-4c0f-882a-7ac207cf1535-kube-api-access-48bsc") pod "network-check-target-xmbgf" (UID: "076f350f-8aef-4c0f-882a-7ac207cf1535") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:38:16.114534 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.114465    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b58gn\" (UniqueName: \"kubernetes.io/projected/9a4ba4d0-a20b-4ed7-9fc7-92bf3943faf6-kube-api-access-b58gn\") pod \"iptables-alerter-29bsx\" (UID: \"9a4ba4d0-a20b-4ed7-9fc7-92bf3943faf6\") " pod="openshift-network-operator/iptables-alerter-29bsx"
Sep 16 12:38:16.115287 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.115258    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jnqk\" (UniqueName: \"kubernetes.io/projected/e5697a20-a56e-4bc9-94c6-73711e6c52ba-kube-api-access-9jnqk\") pod \"multus-l6tn5\" (UID: \"e5697a20-a56e-4bc9-94c6-73711e6c52ba\") " pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:16.115632 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.115615    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pst9f\" (UniqueName: \"kubernetes.io/projected/aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5-kube-api-access-pst9f\") pod \"network-metrics-daemon-tmfpk\" (UID: \"aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5\") " pod="openshift-multus/network-metrics-daemon-tmfpk"
Sep 16 12:38:16.201655 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.201608    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-host-cni-netd\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.201848 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.201660    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d3f21250-f182-428a-a11e-99f2a158ca61-serviceca\") pod \"node-ca-8bxln\" (UID: \"d3f21250-f182-428a-a11e-99f2a158ca61\") " pod="openshift-image-registry/node-ca-8bxln"
Sep 16 12:38:16.201848 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.201691    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d62f767-0859-4bff-a343-9dc4aac3c1f2-var-lib-kubelet\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.201848 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.201720    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a7e49775-cab4-4ccf-a510-9ed41e681e07-cnibin\") pod \"multus-additional-cni-plugins-9f6jr\" (UID: \"a7e49775-cab4-4ccf-a510-9ed41e681e07\") " pod="openshift-multus/multus-additional-cni-plugins-9f6jr"
Sep 16 12:38:16.201848 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.201750    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-host-run-netns\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.201848 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.201779    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-etc-openvswitch\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.202130 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.201809    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.202130 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.201927    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a8b1fe11-76c6-4a4a-91a3-865667440c00-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s6psq\" (UID: \"a8b1fe11-76c6-4a4a-91a3-865667440c00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6psq"
Sep 16 12:38:16.202130 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.201955    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a8b1fe11-76c6-4a4a-91a3-865667440c00-sys-fs\") pod \"aws-ebs-csi-driver-node-s6psq\" (UID: \"a8b1fe11-76c6-4a4a-91a3-865667440c00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6psq"
Sep 16 12:38:16.202130 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.201998    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-systemd-units\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.202130 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.202025    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-host-run-ovn-kubernetes\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.202130 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.202054    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-ovnkube-config\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.202361 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.202087    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5d62f767-0859-4bff-a343-9dc4aac3c1f2-etc-modprobe-d\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.202361 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.202232    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5d62f767-0859-4bff-a343-9dc4aac3c1f2-etc-sysconfig\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.202361 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.202261    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d62f767-0859-4bff-a343-9dc4aac3c1f2-etc-kubernetes\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.202361 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.202292    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqfdl\" (UniqueName: \"kubernetes.io/projected/c44032b0-4e54-42d9-945f-7d401e20d50c-kube-api-access-dqfdl\") pod \"node-resolver-t47mb\" (UID: \"c44032b0-4e54-42d9-945f-7d401e20d50c\") " pod="openshift-dns/node-resolver-t47mb"
Sep 16 12:38:16.202361 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.202320    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjq9p\" (UniqueName: \"kubernetes.io/projected/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-kube-api-access-cjq9p\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.202537 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.202348    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5d62f767-0859-4bff-a343-9dc4aac3c1f2-etc-sysctl-conf\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.202537 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.202407    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5d62f767-0859-4bff-a343-9dc4aac3c1f2-run\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.202537 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.202435    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d62f767-0859-4bff-a343-9dc4aac3c1f2-host\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.202537 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.202467    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5d62f767-0859-4bff-a343-9dc4aac3c1f2-etc-tuned\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.202537 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.202502    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-var-lib-openvswitch\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.202537 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.202529    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d62f767-0859-4bff-a343-9dc4aac3c1f2-sys\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.202814 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.202558    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-node-log\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.202814 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.202586    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a8b1fe11-76c6-4a4a-91a3-865667440c00-device-dir\") pod \"aws-ebs-csi-driver-node-s6psq\" (UID: \"a8b1fe11-76c6-4a4a-91a3-865667440c00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6psq"
Sep 16 12:38:16.202814 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.202615    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpmt6\" (UniqueName: \"kubernetes.io/projected/a8b1fe11-76c6-4a4a-91a3-865667440c00-kube-api-access-jpmt6\") pod \"aws-ebs-csi-driver-node-s6psq\" (UID: \"a8b1fe11-76c6-4a4a-91a3-865667440c00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6psq"
Sep 16 12:38:16.202814 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.202649    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-run-openvswitch\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.202814 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.202681    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a7e49775-cab4-4ccf-a510-9ed41e681e07-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9f6jr\" (UID: \"a7e49775-cab4-4ccf-a510-9ed41e681e07\") " pod="openshift-multus/multus-additional-cni-plugins-9f6jr"
Sep 16 12:38:16.202814 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.202711    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a7e49775-cab4-4ccf-a510-9ed41e681e07-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9f6jr\" (UID: \"a7e49775-cab4-4ccf-a510-9ed41e681e07\") " pod="openshift-multus/multus-additional-cni-plugins-9f6jr"
Sep 16 12:38:16.202814 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.202744    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c44032b0-4e54-42d9-945f-7d401e20d50c-tmp-dir\") pod \"node-resolver-t47mb\" (UID: \"c44032b0-4e54-42d9-945f-7d401e20d50c\") " pod="openshift-dns/node-resolver-t47mb"
Sep 16 12:38:16.202814 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.202771    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-host-cni-bin\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.202814 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.202798    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5d62f767-0859-4bff-a343-9dc4aac3c1f2-etc-sysctl-d\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.203219 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.202826    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5d62f767-0859-4bff-a343-9dc4aac3c1f2-etc-systemd\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.203219 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.202844    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a8b1fe11-76c6-4a4a-91a3-865667440c00-registration-dir\") pod \"aws-ebs-csi-driver-node-s6psq\" (UID: \"a8b1fe11-76c6-4a4a-91a3-865667440c00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6psq"
Sep 16 12:38:16.203219 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.202862    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-run-ovn\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.203219 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.202884    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-env-overrides\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.203219 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.202913    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a8b1fe11-76c6-4a4a-91a3-865667440c00-etc-selinux\") pod \"aws-ebs-csi-driver-node-s6psq\" (UID: \"a8b1fe11-76c6-4a4a-91a3-865667440c00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6psq"
Sep 16 12:38:16.203219 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.202958    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw7ln\" (UniqueName: \"kubernetes.io/projected/a7e49775-cab4-4ccf-a510-9ed41e681e07-kube-api-access-pw7ln\") pod \"multus-additional-cni-plugins-9f6jr\" (UID: \"a7e49775-cab4-4ccf-a510-9ed41e681e07\") " pod="openshift-multus/multus-additional-cni-plugins-9f6jr"
Sep 16 12:38:16.203219 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.202989    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bl6x\" (UniqueName: \"kubernetes.io/projected/d3f21250-f182-428a-a11e-99f2a158ca61-kube-api-access-9bl6x\") pod \"node-ca-8bxln\" (UID: \"d3f21250-f182-428a-a11e-99f2a158ca61\") " pod="openshift-image-registry/node-ca-8bxln"
Sep 16 12:38:16.203219 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.203019    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a8b1fe11-76c6-4a4a-91a3-865667440c00-socket-dir\") pod \"aws-ebs-csi-driver-node-s6psq\" (UID: \"a8b1fe11-76c6-4a4a-91a3-865667440c00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6psq"
Sep 16 12:38:16.203219 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.203062    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a7e49775-cab4-4ccf-a510-9ed41e681e07-system-cni-dir\") pod \"multus-additional-cni-plugins-9f6jr\" (UID: \"a7e49775-cab4-4ccf-a510-9ed41e681e07\") " pod="openshift-multus/multus-additional-cni-plugins-9f6jr"
Sep 16 12:38:16.203219 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.203092    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-ovn-node-metrics-cert\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.203219 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.203139    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d62f767-0859-4bff-a343-9dc4aac3c1f2-tmp\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.203219 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.203190    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-log-socket\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.203219 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.203219    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a7e49775-cab4-4ccf-a510-9ed41e681e07-cni-binary-copy\") pod \"multus-additional-cni-plugins-9f6jr\" (UID: \"a7e49775-cab4-4ccf-a510-9ed41e681e07\") " pod="openshift-multus/multus-additional-cni-plugins-9f6jr"
Sep 16 12:38:16.203758 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.203249    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c44032b0-4e54-42d9-945f-7d401e20d50c-hosts-file\") pod \"node-resolver-t47mb\" (UID: \"c44032b0-4e54-42d9-945f-7d401e20d50c\") " pod="openshift-dns/node-resolver-t47mb"
Sep 16 12:38:16.203758 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.203330    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d62f767-0859-4bff-a343-9dc4aac3c1f2-lib-modules\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.203758 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.203361    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a7e49775-cab4-4ccf-a510-9ed41e681e07-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9f6jr\" (UID: \"a7e49775-cab4-4ccf-a510-9ed41e681e07\") " pod="openshift-multus/multus-additional-cni-plugins-9f6jr"
Sep 16 12:38:16.203758 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.203389    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-host-kubelet\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.203758 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.203423    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-ovnkube-script-lib\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.203758 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.203453    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcb9h\" (UniqueName: \"kubernetes.io/projected/5d62f767-0859-4bff-a343-9dc4aac3c1f2-kube-api-access-jcb9h\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.203758 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.203487    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-run-systemd\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.203758 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.203527    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-host-slash\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.203758 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.203682    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3f21250-f182-428a-a11e-99f2a158ca61-host\") pod \"node-ca-8bxln\" (UID: \"d3f21250-f182-428a-a11e-99f2a158ca61\") " pod="openshift-image-registry/node-ca-8bxln"
Sep 16 12:38:16.204214 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.203813    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3f21250-f182-428a-a11e-99f2a158ca61-host\") pod \"node-ca-8bxln\" (UID: \"d3f21250-f182-428a-a11e-99f2a158ca61\") " pod="openshift-image-registry/node-ca-8bxln"
Sep 16 12:38:16.204214 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.203844    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-run-openvswitch\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.204214 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.203907    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-host-cni-netd\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.204619 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.204552    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d3f21250-f182-428a-a11e-99f2a158ca61-serviceca\") pod \"node-ca-8bxln\" (UID: \"d3f21250-f182-428a-a11e-99f2a158ca61\") " pod="openshift-image-registry/node-ca-8bxln"
Sep 16 12:38:16.204738 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.204641    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d62f767-0859-4bff-a343-9dc4aac3c1f2-var-lib-kubelet\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.204738 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.204684    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a7e49775-cab4-4ccf-a510-9ed41e681e07-cnibin\") pod \"multus-additional-cni-plugins-9f6jr\" (UID: \"a7e49775-cab4-4ccf-a510-9ed41e681e07\") " pod="openshift-multus/multus-additional-cni-plugins-9f6jr"
Sep 16 12:38:16.204738 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.204726    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-host-run-netns\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.204860 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.204768    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-etc-openvswitch\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.204860 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.204815    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.204930 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.204871    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a8b1fe11-76c6-4a4a-91a3-865667440c00-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s6psq\" (UID: \"a8b1fe11-76c6-4a4a-91a3-865667440c00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6psq"
Sep 16 12:38:16.204930 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.204924    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a8b1fe11-76c6-4a4a-91a3-865667440c00-sys-fs\") pod \"aws-ebs-csi-driver-node-s6psq\" (UID: \"a8b1fe11-76c6-4a4a-91a3-865667440c00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6psq"
Sep 16 12:38:16.205004 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.204969    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-systemd-units\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.205052 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.205012    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-host-run-ovn-kubernetes\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.205162 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.205144    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a7e49775-cab4-4ccf-a510-9ed41e681e07-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9f6jr\" (UID: \"a7e49775-cab4-4ccf-a510-9ed41e681e07\") " pod="openshift-multus/multus-additional-cni-plugins-9f6jr"
Sep 16 12:38:16.205295 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.205268    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5d62f767-0859-4bff-a343-9dc4aac3c1f2-run\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.205429 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.205412    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5d62f767-0859-4bff-a343-9dc4aac3c1f2-etc-modprobe-d\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.205475 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.205452    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c44032b0-4e54-42d9-945f-7d401e20d50c-tmp-dir\") pod \"node-resolver-t47mb\" (UID: \"c44032b0-4e54-42d9-945f-7d401e20d50c\") " pod="openshift-dns/node-resolver-t47mb"
Sep 16 12:38:16.205515 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.205477    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5d62f767-0859-4bff-a343-9dc4aac3c1f2-etc-sysconfig\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.205569 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.205532    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d62f767-0859-4bff-a343-9dc4aac3c1f2-etc-kubernetes\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.205569 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.205550    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-host-cni-bin\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.205646 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.205583    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-ovnkube-config\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.205646 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.205638    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d62f767-0859-4bff-a343-9dc4aac3c1f2-host\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.205766 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.205676    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5d62f767-0859-4bff-a343-9dc4aac3c1f2-etc-sysctl-d\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.205766 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.205729    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5d62f767-0859-4bff-a343-9dc4aac3c1f2-etc-systemd\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.205855 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.205781    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a8b1fe11-76c6-4a4a-91a3-865667440c00-registration-dir\") pod \"aws-ebs-csi-driver-node-s6psq\" (UID: \"a8b1fe11-76c6-4a4a-91a3-865667440c00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6psq"
Sep 16 12:38:16.205957 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.205933    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5d62f767-0859-4bff-a343-9dc4aac3c1f2-etc-sysctl-conf\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.206055 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.206029    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-run-ovn\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.206159 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.206138    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-node-log\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.206230 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.206204    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-var-lib-openvswitch\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.206309 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.206264    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d62f767-0859-4bff-a343-9dc4aac3c1f2-sys\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.206391 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.206328    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a8b1fe11-76c6-4a4a-91a3-865667440c00-device-dir\") pod \"aws-ebs-csi-driver-node-s6psq\" (UID: \"a8b1fe11-76c6-4a4a-91a3-865667440c00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6psq"
Sep 16 12:38:16.206487 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.206437    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-env-overrides\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.206578 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.206560    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a8b1fe11-76c6-4a4a-91a3-865667440c00-etc-selinux\") pod \"aws-ebs-csi-driver-node-s6psq\" (UID: \"a8b1fe11-76c6-4a4a-91a3-865667440c00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6psq"
Sep 16 12:38:16.206638 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.206619    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c44032b0-4e54-42d9-945f-7d401e20d50c-hosts-file\") pod \"node-resolver-t47mb\" (UID: \"c44032b0-4e54-42d9-945f-7d401e20d50c\") " pod="openshift-dns/node-resolver-t47mb"
Sep 16 12:38:16.206736 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.206700    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d62f767-0859-4bff-a343-9dc4aac3c1f2-lib-modules\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.207032 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.206850    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a7e49775-cab4-4ccf-a510-9ed41e681e07-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9f6jr\" (UID: \"a7e49775-cab4-4ccf-a510-9ed41e681e07\") " pod="openshift-multus/multus-additional-cni-plugins-9f6jr"
Sep 16 12:38:16.207032 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.206898    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-host-kubelet\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.207157 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.207089    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a7e49775-cab4-4ccf-a510-9ed41e681e07-cni-binary-copy\") pod \"multus-additional-cni-plugins-9f6jr\" (UID: \"a7e49775-cab4-4ccf-a510-9ed41e681e07\") " pod="openshift-multus/multus-additional-cni-plugins-9f6jr"
Sep 16 12:38:16.207458 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.207437    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-ovnkube-script-lib\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.207656 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.207635    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-run-systemd\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.207732 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.207692    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-host-slash\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.207826 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.207805    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a8b1fe11-76c6-4a4a-91a3-865667440c00-socket-dir\") pod \"aws-ebs-csi-driver-node-s6psq\" (UID: \"a8b1fe11-76c6-4a4a-91a3-865667440c00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6psq"
Sep 16 12:38:16.208269 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.208239    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-log-socket\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.208426 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.208268    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a7e49775-cab4-4ccf-a510-9ed41e681e07-system-cni-dir\") pod \"multus-additional-cni-plugins-9f6jr\" (UID: \"a7e49775-cab4-4ccf-a510-9ed41e681e07\") " pod="openshift-multus/multus-additional-cni-plugins-9f6jr"
Sep 16 12:38:16.208735 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.208713    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a7e49775-cab4-4ccf-a510-9ed41e681e07-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9f6jr\" (UID: \"a7e49775-cab4-4ccf-a510-9ed41e681e07\") " pod="openshift-multus/multus-additional-cni-plugins-9f6jr"
Sep 16 12:38:16.210994 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.210975    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d62f767-0859-4bff-a343-9dc4aac3c1f2-tmp\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.211176 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.211036    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5d62f767-0859-4bff-a343-9dc4aac3c1f2-etc-tuned\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.211847 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.211784    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-ovn-node-metrics-cert\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.225228 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.225180    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpmt6\" (UniqueName: \"kubernetes.io/projected/a8b1fe11-76c6-4a4a-91a3-865667440c00-kube-api-access-jpmt6\") pod \"aws-ebs-csi-driver-node-s6psq\" (UID: \"a8b1fe11-76c6-4a4a-91a3-865667440c00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6psq"
Sep 16 12:38:16.225773 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.225372    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqfdl\" (UniqueName: \"kubernetes.io/projected/c44032b0-4e54-42d9-945f-7d401e20d50c-kube-api-access-dqfdl\") pod \"node-resolver-t47mb\" (UID: \"c44032b0-4e54-42d9-945f-7d401e20d50c\") " pod="openshift-dns/node-resolver-t47mb"
Sep 16 12:38:16.225773 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.225535    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw7ln\" (UniqueName: \"kubernetes.io/projected/a7e49775-cab4-4ccf-a510-9ed41e681e07-kube-api-access-pw7ln\") pod \"multus-additional-cni-plugins-9f6jr\" (UID: \"a7e49775-cab4-4ccf-a510-9ed41e681e07\") " pod="openshift-multus/multus-additional-cni-plugins-9f6jr"
Sep 16 12:38:16.225773 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.225733    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcb9h\" (UniqueName: \"kubernetes.io/projected/5d62f767-0859-4bff-a343-9dc4aac3c1f2-kube-api-access-jcb9h\") pod \"tuned-ftpbx\" (UID: \"5d62f767-0859-4bff-a343-9dc4aac3c1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.228992 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.228965    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bl6x\" (UniqueName: \"kubernetes.io/projected/d3f21250-f182-428a-a11e-99f2a158ca61-kube-api-access-9bl6x\") pod \"node-ca-8bxln\" (UID: \"d3f21250-f182-428a-a11e-99f2a158ca61\") " pod="openshift-image-registry/node-ca-8bxln"
Sep 16 12:38:16.231483 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.231455    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjq9p\" (UniqueName: \"kubernetes.io/projected/e4f4a5d3-fc62-4048-a18f-3ef216a53a9f-kube-api-access-cjq9p\") pod \"ovnkube-node-6g82p\" (UID: \"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.288274 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.288240    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j2kjn"
Sep 16 12:38:16.302012 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.301987    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l6tn5"
Sep 16 12:38:16.311109 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.311087    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-29bsx"
Sep 16 12:38:16.316102 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.316074    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6psq"
Sep 16 12:38:16.326002 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.325977    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8bxln"
Sep 16 12:38:16.335895 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.335871    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9f6jr"
Sep 16 12:38:16.345750 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.345729    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t47mb"
Sep 16 12:38:16.355672 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.355646    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ftpbx"
Sep 16 12:38:16.363539 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.363522    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:16.607993 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.606945    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/342f831e-0ffe-4582-b616-e0ce14364b32-original-pull-secret\") pod \"global-pull-secret-syncer-bwb7l\" (UID: \"342f831e-0ffe-4582-b616-e0ce14364b32\") " pod="kube-system/global-pull-secret-syncer-bwb7l"
Sep 16 12:38:16.607993 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.607001    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5-metrics-certs\") pod \"network-metrics-daemon-tmfpk\" (UID: \"aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5\") " pod="openshift-multus/network-metrics-daemon-tmfpk"
Sep 16 12:38:16.607993 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.607078    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48bsc\" (UniqueName: \"kubernetes.io/projected/076f350f-8aef-4c0f-882a-7ac207cf1535-kube-api-access-48bsc\") pod \"network-check-target-xmbgf\" (UID: \"076f350f-8aef-4c0f-882a-7ac207cf1535\") " pod="openshift-network-diagnostics/network-check-target-xmbgf"
Sep 16 12:38:16.607993 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:16.607222    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:38:16.607993 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:16.607240    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:38:16.607993 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:16.607254    2577 projected.go:196] Error preparing data for projected volume kube-api-access-48bsc for pod openshift-network-diagnostics/network-check-target-xmbgf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:38:16.607993 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:16.607314    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/076f350f-8aef-4c0f-882a-7ac207cf1535-kube-api-access-48bsc podName:076f350f-8aef-4c0f-882a-7ac207cf1535 nodeName:}" failed. No retries permitted until 2026-09-16 12:38:17.607293745 +0000 UTC m=+4.054526368 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-48bsc" (UniqueName: "kubernetes.io/projected/076f350f-8aef-4c0f-882a-7ac207cf1535-kube-api-access-48bsc") pod "network-check-target-xmbgf" (UID: "076f350f-8aef-4c0f-882a-7ac207cf1535") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:38:16.607993 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:16.607764    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:38:16.607993 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:16.607816    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/342f831e-0ffe-4582-b616-e0ce14364b32-original-pull-secret podName:342f831e-0ffe-4582-b616-e0ce14364b32 nodeName:}" failed. No retries permitted until 2026-09-16 12:38:17.607796357 +0000 UTC m=+4.055028979 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/342f831e-0ffe-4582-b616-e0ce14364b32-original-pull-secret") pod "global-pull-secret-syncer-bwb7l" (UID: "342f831e-0ffe-4582-b616-e0ce14364b32") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:38:16.607993 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:16.607875    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:38:16.607993 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:16.607910    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5-metrics-certs podName:aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5 nodeName:}" failed. No retries permitted until 2026-09-16 12:38:17.607896989 +0000 UTC m=+4.055129620 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5-metrics-certs") pod "network-metrics-daemon-tmfpk" (UID: "aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:38:16.662894 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:16.662864    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3f21250_f182_428a_a11e_99f2a158ca61.slice/crio-d4b5fe535a3f6bde968df2f08518e4720ce82f712006453207842e6b3a211872 WatchSource:0}: Error finding container d4b5fe535a3f6bde968df2f08518e4720ce82f712006453207842e6b3a211872: Status 404 returned error can't find the container with id d4b5fe535a3f6bde968df2f08518e4720ce82f712006453207842e6b3a211872
Sep 16 12:38:16.664101 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:16.664067    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d62f767_0859_4bff_a343_9dc4aac3c1f2.slice/crio-20b64bc69cbde8de3c17de67e94b4437eddf4466be71830ea524196af49ca690 WatchSource:0}: Error finding container 20b64bc69cbde8de3c17de67e94b4437eddf4466be71830ea524196af49ca690: Status 404 returned error can't find the container with id 20b64bc69cbde8de3c17de67e94b4437eddf4466be71830ea524196af49ca690
Sep 16 12:38:16.665693 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:16.665667    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9fcc2dc_c3c3_401c_8534_95cd3908332e.slice/crio-d5153eb6b06741b43188926a59faddef74a72dad5907fb7477c634efefbe10df WatchSource:0}: Error finding container d5153eb6b06741b43188926a59faddef74a72dad5907fb7477c634efefbe10df: Status 404 returned error can't find the container with id d5153eb6b06741b43188926a59faddef74a72dad5907fb7477c634efefbe10df
Sep 16 12:38:16.666910 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:16.666889    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc44032b0_4e54_42d9_945f_7d401e20d50c.slice/crio-9b12609c539aa981047de73ae59828622661910558a55d2c7da3c35a9089cdbe WatchSource:0}: Error finding container 9b12609c539aa981047de73ae59828622661910558a55d2c7da3c35a9089cdbe: Status 404 returned error can't find the container with id 9b12609c539aa981047de73ae59828622661910558a55d2c7da3c35a9089cdbe
Sep 16 12:38:16.670687 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:16.670668    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a4ba4d0_a20b_4ed7_9fc7_92bf3943faf6.slice/crio-c23e982889389363e8747c3138032d17997990c2fb975eb5277b57ebad6123b8 WatchSource:0}: Error finding container c23e982889389363e8747c3138032d17997990c2fb975eb5277b57ebad6123b8: Status 404 returned error can't find the container with id c23e982889389363e8747c3138032d17997990c2fb975eb5277b57ebad6123b8
Sep 16 12:38:16.672221 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:16.672199    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7e49775_cab4_4ccf_a510_9ed41e681e07.slice/crio-e784ee0f62bb09535af453072c07bb010b7b9f10355dcef2f11a76fa23db0a3d WatchSource:0}: Error finding container e784ee0f62bb09535af453072c07bb010b7b9f10355dcef2f11a76fa23db0a3d: Status 404 returned error can't find the container with id e784ee0f62bb09535af453072c07bb010b7b9f10355dcef2f11a76fa23db0a3d
Sep 16 12:38:16.673165 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:16.673144    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5697a20_a56e_4bc9_94c6_73711e6c52ba.slice/crio-0ec0da602d1831977d99cd62930269af3600e61f61283ac6da6bf11e225c9331 WatchSource:0}: Error finding container 0ec0da602d1831977d99cd62930269af3600e61f61283ac6da6bf11e225c9331: Status 404 returned error can't find the container with id 0ec0da602d1831977d99cd62930269af3600e61f61283ac6da6bf11e225c9331
Sep 16 12:38:16.698192 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:16.698161    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8b1fe11_76c6_4a4a_91a3_865667440c00.slice/crio-4cc45a40aa810aaffd961695610a6345a3a66d7d43d7480799e2fd2b9783dc28 WatchSource:0}: Error finding container 4cc45a40aa810aaffd961695610a6345a3a66d7d43d7480799e2fd2b9783dc28: Status 404 returned error can't find the container with id 4cc45a40aa810aaffd961695610a6345a3a66d7d43d7480799e2fd2b9783dc28
Sep 16 12:38:16.698920 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:16.698739    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f4a5d3_fc62_4048_a18f_3ef216a53a9f.slice/crio-79f052dbcee15eb9e6986086f0bded8f39c9a9e5d3593b77fd31089da9d9f938 WatchSource:0}: Error finding container 79f052dbcee15eb9e6986086f0bded8f39c9a9e5d3593b77fd31089da9d9f938: Status 404 returned error can't find the container with id 79f052dbcee15eb9e6986086f0bded8f39c9a9e5d3593b77fd31089da9d9f938
Sep 16 12:38:16.919854 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:16.919759    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:38:17.047639 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:17.047594    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 12:33:16 +0000 UTC" deadline="2028-04-05 12:00:50.512653234 +0000 UTC"
Sep 16 12:38:17.047639 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:17.047638    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13607h22m33.465018405s"
Sep 16 12:38:17.060522 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:17.060487    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ftpbx" event={"ID":"5d62f767-0859-4bff-a343-9dc4aac3c1f2","Type":"ContainerStarted","Data":"20b64bc69cbde8de3c17de67e94b4437eddf4466be71830ea524196af49ca690"}
Sep 16 12:38:17.061859 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:17.061830    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6psq" event={"ID":"a8b1fe11-76c6-4a4a-91a3-865667440c00","Type":"ContainerStarted","Data":"4cc45a40aa810aaffd961695610a6345a3a66d7d43d7480799e2fd2b9783dc28"}
Sep 16 12:38:17.063391 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:17.063369    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9f6jr" event={"ID":"a7e49775-cab4-4ccf-a510-9ed41e681e07","Type":"ContainerStarted","Data":"e784ee0f62bb09535af453072c07bb010b7b9f10355dcef2f11a76fa23db0a3d"}
Sep 16 12:38:17.064872 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:17.064833    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-29bsx" event={"ID":"9a4ba4d0-a20b-4ed7-9fc7-92bf3943faf6","Type":"ContainerStarted","Data":"c23e982889389363e8747c3138032d17997990c2fb975eb5277b57ebad6123b8"}
Sep 16 12:38:17.066352 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:17.066315    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t47mb" event={"ID":"c44032b0-4e54-42d9-945f-7d401e20d50c","Type":"ContainerStarted","Data":"9b12609c539aa981047de73ae59828622661910558a55d2c7da3c35a9089cdbe"}
Sep 16 12:38:17.067786 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:17.067761    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j2kjn" event={"ID":"c9fcc2dc-c3c3-401c-8534-95cd3908332e","Type":"ContainerStarted","Data":"d5153eb6b06741b43188926a59faddef74a72dad5907fb7477c634efefbe10df"}
Sep 16 12:38:17.069283 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:17.069258    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8bxln" event={"ID":"d3f21250-f182-428a-a11e-99f2a158ca61","Type":"ContainerStarted","Data":"d4b5fe535a3f6bde968df2f08518e4720ce82f712006453207842e6b3a211872"}
Sep 16 12:38:17.071626 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:17.071589    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-190.ec2.internal" event={"ID":"71c783a701963269184ce00c0e4aab2a","Type":"ContainerStarted","Data":"3ee4458633d6137365f9fe0d266767a2f36c03882975389233c00ecab0b7fa6f"}
Sep 16 12:38:17.073205 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:17.073175    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6g82p" event={"ID":"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f","Type":"ContainerStarted","Data":"79f052dbcee15eb9e6986086f0bded8f39c9a9e5d3593b77fd31089da9d9f938"}
Sep 16 12:38:17.074565 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:17.074540    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l6tn5" event={"ID":"e5697a20-a56e-4bc9-94c6-73711e6c52ba","Type":"ContainerStarted","Data":"0ec0da602d1831977d99cd62930269af3600e61f61283ac6da6bf11e225c9331"}
Sep 16 12:38:17.616152 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:17.616049    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48bsc\" (UniqueName: \"kubernetes.io/projected/076f350f-8aef-4c0f-882a-7ac207cf1535-kube-api-access-48bsc\") pod \"network-check-target-xmbgf\" (UID: \"076f350f-8aef-4c0f-882a-7ac207cf1535\") " pod="openshift-network-diagnostics/network-check-target-xmbgf"
Sep 16 12:38:17.616152 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:17.616143    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/342f831e-0ffe-4582-b616-e0ce14364b32-original-pull-secret\") pod \"global-pull-secret-syncer-bwb7l\" (UID: \"342f831e-0ffe-4582-b616-e0ce14364b32\") " pod="kube-system/global-pull-secret-syncer-bwb7l"
Sep 16 12:38:17.616366 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:17.616179    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5-metrics-certs\") pod \"network-metrics-daemon-tmfpk\" (UID: \"aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5\") " pod="openshift-multus/network-metrics-daemon-tmfpk"
Sep 16 12:38:17.616366 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:17.616315    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:38:17.616483 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:17.616380    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5-metrics-certs podName:aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5 nodeName:}" failed. No retries permitted until 2026-09-16 12:38:19.616360048 +0000 UTC m=+6.063592672 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5-metrics-certs") pod "network-metrics-daemon-tmfpk" (UID: "aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:38:17.616483 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:17.616474    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:38:17.616601 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:17.616492    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:38:17.616601 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:17.616507    2577 projected.go:196] Error preparing data for projected volume kube-api-access-48bsc for pod openshift-network-diagnostics/network-check-target-xmbgf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:38:17.616601 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:17.616555    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/076f350f-8aef-4c0f-882a-7ac207cf1535-kube-api-access-48bsc podName:076f350f-8aef-4c0f-882a-7ac207cf1535 nodeName:}" failed. No retries permitted until 2026-09-16 12:38:19.616543238 +0000 UTC m=+6.063775877 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-48bsc" (UniqueName: "kubernetes.io/projected/076f350f-8aef-4c0f-882a-7ac207cf1535-kube-api-access-48bsc") pod "network-check-target-xmbgf" (UID: "076f350f-8aef-4c0f-882a-7ac207cf1535") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:38:17.616757 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:17.616619    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:38:17.616757 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:17.616652    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/342f831e-0ffe-4582-b616-e0ce14364b32-original-pull-secret podName:342f831e-0ffe-4582-b616-e0ce14364b32 nodeName:}" failed. No retries permitted until 2026-09-16 12:38:19.616640482 +0000 UTC m=+6.063873107 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/342f831e-0ffe-4582-b616-e0ce14364b32-original-pull-secret") pod "global-pull-secret-syncer-bwb7l" (UID: "342f831e-0ffe-4582-b616-e0ce14364b32") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:38:18.054412 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:18.050871    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwb7l"
Sep 16 12:38:18.054412 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:18.051247    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwb7l" podUID="342f831e-0ffe-4582-b616-e0ce14364b32"
Sep 16 12:38:18.054412 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:18.051312    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 12:33:16 +0000 UTC" deadline="2028-06-09 21:09:53.129305066 +0000 UTC"
Sep 16 12:38:18.054412 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:18.051333    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15176h31m35.077975964s"
Sep 16 12:38:18.054412 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:18.051412    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmfpk"
Sep 16 12:38:18.054412 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:18.051499    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmfpk" podUID="aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5"
Sep 16 12:38:18.054412 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:18.051562    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmbgf"
Sep 16 12:38:18.054412 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:18.051630    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmbgf" podUID="076f350f-8aef-4c0f-882a-7ac207cf1535"
Sep 16 12:38:18.086600 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:18.085846    2577 generic.go:356] "Generic (PLEG): container finished" podID="71c783a701963269184ce00c0e4aab2a" containerID="3ee4458633d6137365f9fe0d266767a2f36c03882975389233c00ecab0b7fa6f" exitCode=0
Sep 16 12:38:18.086600 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:18.085937    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-190.ec2.internal" event={"ID":"71c783a701963269184ce00c0e4aab2a","Type":"ContainerDied","Data":"3ee4458633d6137365f9fe0d266767a2f36c03882975389233c00ecab0b7fa6f"}
Sep 16 12:38:18.092441 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:18.091139    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-190.ec2.internal" event={"ID":"dad4cbabbaf3126a41405ddf99389dd4","Type":"ContainerStarted","Data":"a70b3df5e8167dbec59498e8f42001a22eeaff6e4d96475d5ded507729fe911b"}
Sep 16 12:38:19.097911 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:19.097869    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-190.ec2.internal" event={"ID":"71c783a701963269184ce00c0e4aab2a","Type":"ContainerStarted","Data":"e5a1a18f18008fa91188cc0f37d3bb077e768abd6484e6a4de51bc5497a1da95"}
Sep 16 12:38:19.116715 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:19.116646    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-190.ec2.internal" podStartSLOduration=3.116622613 podStartE2EDuration="3.116622613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:38:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:38:18.122081997 +0000 UTC m=+4.569314646" watchObservedRunningTime="2026-09-16 12:38:19.116622613 +0000 UTC m=+5.563855247"
Sep 16 12:38:19.633513 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:19.633466    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48bsc\" (UniqueName: \"kubernetes.io/projected/076f350f-8aef-4c0f-882a-7ac207cf1535-kube-api-access-48bsc\") pod \"network-check-target-xmbgf\" (UID: \"076f350f-8aef-4c0f-882a-7ac207cf1535\") " pod="openshift-network-diagnostics/network-check-target-xmbgf"
Sep 16 12:38:19.633701 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:19.633545    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/342f831e-0ffe-4582-b616-e0ce14364b32-original-pull-secret\") pod \"global-pull-secret-syncer-bwb7l\" (UID: \"342f831e-0ffe-4582-b616-e0ce14364b32\") " pod="kube-system/global-pull-secret-syncer-bwb7l"
Sep 16 12:38:19.633701 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:19.633583    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5-metrics-certs\") pod \"network-metrics-daemon-tmfpk\" (UID: \"aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5\") " pod="openshift-multus/network-metrics-daemon-tmfpk"
Sep 16 12:38:19.633815 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:19.633727    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:38:19.633815 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:19.633793    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5-metrics-certs podName:aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5 nodeName:}" failed. No retries permitted until 2026-09-16 12:38:23.633772974 +0000 UTC m=+10.081005600 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5-metrics-certs") pod "network-metrics-daemon-tmfpk" (UID: "aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:38:19.634318 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:19.634291    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:38:19.634438 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:19.634325    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:38:19.634438 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:19.634339    2577 projected.go:196] Error preparing data for projected volume kube-api-access-48bsc for pod openshift-network-diagnostics/network-check-target-xmbgf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:38:19.634438 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:19.634390    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/076f350f-8aef-4c0f-882a-7ac207cf1535-kube-api-access-48bsc podName:076f350f-8aef-4c0f-882a-7ac207cf1535 nodeName:}" failed. No retries permitted until 2026-09-16 12:38:23.63437319 +0000 UTC m=+10.081605815 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-48bsc" (UniqueName: "kubernetes.io/projected/076f350f-8aef-4c0f-882a-7ac207cf1535-kube-api-access-48bsc") pod "network-check-target-xmbgf" (UID: "076f350f-8aef-4c0f-882a-7ac207cf1535") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:38:19.634618 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:19.634454    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:38:19.634618 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:19.634493    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/342f831e-0ffe-4582-b616-e0ce14364b32-original-pull-secret podName:342f831e-0ffe-4582-b616-e0ce14364b32 nodeName:}" failed. No retries permitted until 2026-09-16 12:38:23.63448176 +0000 UTC m=+10.081714382 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/342f831e-0ffe-4582-b616-e0ce14364b32-original-pull-secret") pod "global-pull-secret-syncer-bwb7l" (UID: "342f831e-0ffe-4582-b616-e0ce14364b32") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:38:20.050044 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:20.049248    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmbgf"
Sep 16 12:38:20.050044 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:20.049387    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmbgf" podUID="076f350f-8aef-4c0f-882a-7ac207cf1535"
Sep 16 12:38:20.050044 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:20.049837    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwb7l"
Sep 16 12:38:20.050044 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:20.049944    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwb7l" podUID="342f831e-0ffe-4582-b616-e0ce14364b32"
Sep 16 12:38:20.050772 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:20.050561    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmfpk"
Sep 16 12:38:20.050772 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:20.050684    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmfpk" podUID="aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5"
Sep 16 12:38:22.051950 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:22.051146    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwb7l"
Sep 16 12:38:22.051950 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:22.051280    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwb7l" podUID="342f831e-0ffe-4582-b616-e0ce14364b32"
Sep 16 12:38:22.051950 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:22.051692    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmfpk"
Sep 16 12:38:22.051950 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:22.051778    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmfpk" podUID="aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5"
Sep 16 12:38:22.051950 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:22.051831    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmbgf"
Sep 16 12:38:22.051950 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:22.051903    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmbgf" podUID="076f350f-8aef-4c0f-882a-7ac207cf1535"
Sep 16 12:38:23.671402 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:23.671354    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48bsc\" (UniqueName: \"kubernetes.io/projected/076f350f-8aef-4c0f-882a-7ac207cf1535-kube-api-access-48bsc\") pod \"network-check-target-xmbgf\" (UID: \"076f350f-8aef-4c0f-882a-7ac207cf1535\") " pod="openshift-network-diagnostics/network-check-target-xmbgf"
Sep 16 12:38:23.671974 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:23.671424    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/342f831e-0ffe-4582-b616-e0ce14364b32-original-pull-secret\") pod \"global-pull-secret-syncer-bwb7l\" (UID: \"342f831e-0ffe-4582-b616-e0ce14364b32\") " pod="kube-system/global-pull-secret-syncer-bwb7l"
Sep 16 12:38:23.671974 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:23.671465    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5-metrics-certs\") pod \"network-metrics-daemon-tmfpk\" (UID: \"aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5\") " pod="openshift-multus/network-metrics-daemon-tmfpk"
Sep 16 12:38:23.671974 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:23.671684    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:38:23.671974 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:23.671759    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5-metrics-certs podName:aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5 nodeName:}" failed. No retries permitted until 2026-09-16 12:38:31.671734883 +0000 UTC m=+18.118967505 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5-metrics-certs") pod "network-metrics-daemon-tmfpk" (UID: "aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:38:23.672270 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:23.672244    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:38:23.672356 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:23.672329    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/342f831e-0ffe-4582-b616-e0ce14364b32-original-pull-secret podName:342f831e-0ffe-4582-b616-e0ce14364b32 nodeName:}" failed. No retries permitted until 2026-09-16 12:38:31.672306409 +0000 UTC m=+18.119539035 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/342f831e-0ffe-4582-b616-e0ce14364b32-original-pull-secret") pod "global-pull-secret-syncer-bwb7l" (UID: "342f831e-0ffe-4582-b616-e0ce14364b32") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:38:23.672356 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:23.672255    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:38:23.672472 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:23.672369    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:38:23.672472 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:23.672386    2577 projected.go:196] Error preparing data for projected volume kube-api-access-48bsc for pod openshift-network-diagnostics/network-check-target-xmbgf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:38:23.672472 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:23.672451    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/076f350f-8aef-4c0f-882a-7ac207cf1535-kube-api-access-48bsc podName:076f350f-8aef-4c0f-882a-7ac207cf1535 nodeName:}" failed. No retries permitted until 2026-09-16 12:38:31.672434203 +0000 UTC m=+18.119666829 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-48bsc" (UniqueName: "kubernetes.io/projected/076f350f-8aef-4c0f-882a-7ac207cf1535-kube-api-access-48bsc") pod "network-check-target-xmbgf" (UID: "076f350f-8aef-4c0f-882a-7ac207cf1535") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:38:24.049997 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:24.049920    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmbgf"
Sep 16 12:38:24.050168 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:24.050008    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwb7l"
Sep 16 12:38:24.050168 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:24.050029    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmbgf" podUID="076f350f-8aef-4c0f-882a-7ac207cf1535"
Sep 16 12:38:24.050168 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:24.050146    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwb7l" podUID="342f831e-0ffe-4582-b616-e0ce14364b32"
Sep 16 12:38:24.050326 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:24.050221    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmfpk"
Sep 16 12:38:24.050389 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:24.050326    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmfpk" podUID="aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5"
Sep 16 12:38:26.048928 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:26.048887    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmbgf"
Sep 16 12:38:26.048928 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:26.048936    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwb7l"
Sep 16 12:38:26.049498 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:26.049036    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmbgf" podUID="076f350f-8aef-4c0f-882a-7ac207cf1535"
Sep 16 12:38:26.049498 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:26.049159    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwb7l" podUID="342f831e-0ffe-4582-b616-e0ce14364b32"
Sep 16 12:38:26.049498 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:26.049259    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmfpk"
Sep 16 12:38:26.049498 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:26.049364    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmfpk" podUID="aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5"
Sep 16 12:38:28.048544 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:28.048501    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwb7l"
Sep 16 12:38:28.048906 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:28.048555    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmbgf"
Sep 16 12:38:28.048906 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:28.048501    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmfpk"
Sep 16 12:38:28.048906 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:28.048645    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwb7l" podUID="342f831e-0ffe-4582-b616-e0ce14364b32"
Sep 16 12:38:28.048906 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:28.048782    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmfpk" podUID="aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5"
Sep 16 12:38:28.048906 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:28.048853    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmbgf" podUID="076f350f-8aef-4c0f-882a-7ac207cf1535"
Sep 16 12:38:30.049401 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:30.048949    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwb7l"
Sep 16 12:38:30.049401 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:30.049090    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwb7l" podUID="342f831e-0ffe-4582-b616-e0ce14364b32"
Sep 16 12:38:30.049401 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:30.048955    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmfpk"
Sep 16 12:38:30.049401 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:30.049151    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmbgf"
Sep 16 12:38:30.049401 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:30.049216    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmfpk" podUID="aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5"
Sep 16 12:38:30.049401 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:30.049285    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmbgf" podUID="076f350f-8aef-4c0f-882a-7ac207cf1535"
Sep 16 12:38:31.737790 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:31.737755    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48bsc\" (UniqueName: \"kubernetes.io/projected/076f350f-8aef-4c0f-882a-7ac207cf1535-kube-api-access-48bsc\") pod \"network-check-target-xmbgf\" (UID: \"076f350f-8aef-4c0f-882a-7ac207cf1535\") " pod="openshift-network-diagnostics/network-check-target-xmbgf"
Sep 16 12:38:31.738302 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:31.737814    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/342f831e-0ffe-4582-b616-e0ce14364b32-original-pull-secret\") pod \"global-pull-secret-syncer-bwb7l\" (UID: \"342f831e-0ffe-4582-b616-e0ce14364b32\") " pod="kube-system/global-pull-secret-syncer-bwb7l"
Sep 16 12:38:31.738302 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:31.737851    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5-metrics-certs\") pod \"network-metrics-daemon-tmfpk\" (UID: \"aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5\") " pod="openshift-multus/network-metrics-daemon-tmfpk"
Sep 16 12:38:31.738302 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:31.737937    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:38:31.738302 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:31.737983    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:38:31.738302 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:31.737987    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:38:31.738302 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:31.738007    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/342f831e-0ffe-4582-b616-e0ce14364b32-original-pull-secret podName:342f831e-0ffe-4582-b616-e0ce14364b32 nodeName:}" failed. No retries permitted until 2026-09-16 12:38:47.73799127 +0000 UTC m=+34.185223894 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/342f831e-0ffe-4582-b616-e0ce14364b32-original-pull-secret") pod "global-pull-secret-syncer-bwb7l" (UID: "342f831e-0ffe-4582-b616-e0ce14364b32") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:38:31.738302 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:31.738021    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:38:31.738302 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:31.738057    2577 projected.go:196] Error preparing data for projected volume kube-api-access-48bsc for pod openshift-network-diagnostics/network-check-target-xmbgf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:38:31.738302 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:31.738059    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5-metrics-certs podName:aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5 nodeName:}" failed. No retries permitted until 2026-09-16 12:38:47.73802307 +0000 UTC m=+34.185255693 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5-metrics-certs") pod "network-metrics-daemon-tmfpk" (UID: "aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:38:31.738302 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:31.738115    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/076f350f-8aef-4c0f-882a-7ac207cf1535-kube-api-access-48bsc podName:076f350f-8aef-4c0f-882a-7ac207cf1535 nodeName:}" failed. No retries permitted until 2026-09-16 12:38:47.738098478 +0000 UTC m=+34.185331121 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-48bsc" (UniqueName: "kubernetes.io/projected/076f350f-8aef-4c0f-882a-7ac207cf1535-kube-api-access-48bsc") pod "network-check-target-xmbgf" (UID: "076f350f-8aef-4c0f-882a-7ac207cf1535") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:38:32.049206 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:32.049124    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwb7l"
Sep 16 12:38:32.049206 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:32.049169    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmbgf"
Sep 16 12:38:32.049436 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:32.049130    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmfpk"
Sep 16 12:38:32.049436 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:32.049251    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwb7l" podUID="342f831e-0ffe-4582-b616-e0ce14364b32"
Sep 16 12:38:32.049436 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:32.049350    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmfpk" podUID="aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5"
Sep 16 12:38:32.049582 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:32.049436    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmbgf" podUID="076f350f-8aef-4c0f-882a-7ac207cf1535"
Sep 16 12:38:34.049857 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:34.049830    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmfpk"
Sep 16 12:38:34.050349 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:34.049930    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmfpk" podUID="aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5"
Sep 16 12:38:34.050349 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:34.050058    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmbgf"
Sep 16 12:38:34.050349 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:34.050162    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmbgf" podUID="076f350f-8aef-4c0f-882a-7ac207cf1535"
Sep 16 12:38:34.050349 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:34.050177    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwb7l"
Sep 16 12:38:34.050349 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:34.050249    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwb7l" podUID="342f831e-0ffe-4582-b616-e0ce14364b32"
Sep 16 12:38:36.048941 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:36.048707    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmbgf"
Sep 16 12:38:36.049547 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:36.049013    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmbgf" podUID="076f350f-8aef-4c0f-882a-7ac207cf1535"
Sep 16 12:38:36.049547 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:36.048867    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmfpk"
Sep 16 12:38:36.049547 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:36.049159    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmfpk" podUID="aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5"
Sep 16 12:38:36.049547 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:36.048790    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwb7l"
Sep 16 12:38:36.049547 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:36.049256    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwb7l" podUID="342f831e-0ffe-4582-b616-e0ce14364b32"
Sep 16 12:38:36.138892 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:36.138864    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j2kjn" event={"ID":"c9fcc2dc-c3c3-401c-8534-95cd3908332e","Type":"ContainerStarted","Data":"154bac3f22643a1b83faa4fc2f2dc4e3288015375476e46281f176dc87e10132"}
Sep 16 12:38:36.140579 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:36.140553    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8bxln" event={"ID":"d3f21250-f182-428a-a11e-99f2a158ca61","Type":"ContainerStarted","Data":"99708f57aa7c91207e258b3d020065eb8f0014ed4907855f175007119c6fd7c5"}
Sep 16 12:38:36.146669 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:36.146585    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6g82p" event={"ID":"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f","Type":"ContainerStarted","Data":"f768ab54ada6ec30425e5b421375fca2e03e859a686aa0bee78491a2a23b08ad"}
Sep 16 12:38:36.146812 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:36.146731    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6g82p" event={"ID":"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f","Type":"ContainerStarted","Data":"c4d1d304a939474befc1f4c8def9858e87f082b4e3c97078fa3b8fbdf6c17010"}
Sep 16 12:38:36.149108 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:36.148622    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l6tn5" event={"ID":"e5697a20-a56e-4bc9-94c6-73711e6c52ba","Type":"ContainerStarted","Data":"1440dd133e0ee7770e670dd101975a1065f39f248ecd2c069bdbf2d7acbb5987"}
Sep 16 12:38:36.150826 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:36.150803    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ftpbx" event={"ID":"5d62f767-0859-4bff-a343-9dc4aac3c1f2","Type":"ContainerStarted","Data":"a03a17226e9d560c83f5f7a1a84c6bbed52814e95cf2cb48ed7979a3eaa8e717"}
Sep 16 12:38:36.152436 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:36.152416    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6psq" event={"ID":"a8b1fe11-76c6-4a4a-91a3-865667440c00","Type":"ContainerStarted","Data":"1d4420498956b9d49909337e23ea6335b41f3c9712718724a5e4e1feb5b3527b"}
Sep 16 12:38:36.154300 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:36.154260    2577 generic.go:356] "Generic (PLEG): container finished" podID="a7e49775-cab4-4ccf-a510-9ed41e681e07" containerID="2c4a757b1d0fbe02d5ee1179fbdabd1e73766874fae44fdf0716ca452f3fda9f" exitCode=0
Sep 16 12:38:36.154393 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:36.154345    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9f6jr" event={"ID":"a7e49775-cab4-4ccf-a510-9ed41e681e07","Type":"ContainerDied","Data":"2c4a757b1d0fbe02d5ee1179fbdabd1e73766874fae44fdf0716ca452f3fda9f"}
Sep 16 12:38:36.154537 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:36.154488    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-190.ec2.internal" podStartSLOduration=20.154469824 podStartE2EDuration="20.154469824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:38:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:38:19.117310558 +0000 UTC m=+5.564543204" watchObservedRunningTime="2026-09-16 12:38:36.154469824 +0000 UTC m=+22.601702467"
Sep 16 12:38:36.154829 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:36.154791    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-j2kjn" podStartSLOduration=2.018520786 podStartE2EDuration="21.154783308s" totalImagesPullingTime="19.136262522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:38:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:38:36.153799183 +0000 UTC m=+22.601031826" watchObservedRunningTime="2026-09-16 12:38:36.154783308 +0000 UTC m=+22.602016014"
Sep 16 12:38:36.156399 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:36.156172    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t47mb" event={"ID":"c44032b0-4e54-42d9-945f-7d401e20d50c","Type":"ContainerStarted","Data":"1edcf68f88154bc3b215475d2abe6b9a9fd9611b735027e23efeb227d1a17e26"}
Sep 16 12:38:36.170967 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:36.170918    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8bxln" podStartSLOduration=2.022519459 podStartE2EDuration="21.170900249s" totalImagesPullingTime="19.14838079s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:38:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:38:36.170155572 +0000 UTC m=+22.617388214" watchObservedRunningTime="2026-09-16 12:38:36.170900249 +0000 UTC m=+22.618132893"
Sep 16 12:38:36.185405 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:36.185356    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ftpbx" podStartSLOduration=2.047861112 podStartE2EDuration="21.185337415s" totalImagesPullingTime="19.137476303s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:38:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:38:36.185197491 +0000 UTC m=+22.632430138" watchObservedRunningTime="2026-09-16 12:38:36.185337415 +0000 UTC m=+22.632570058"
Sep 16 12:38:36.201819 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:36.201780    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-l6tn5" podStartSLOduration=2.032832737 podStartE2EDuration="21.201766763s" totalImagesPullingTime="19.168934026s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:38:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:38:36.201578793 +0000 UTC m=+22.648811435" watchObservedRunningTime="2026-09-16 12:38:36.201766763 +0000 UTC m=+22.648999406"
Sep 16 12:38:36.254018 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:36.253977    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-t47mb" podStartSLOduration=2.104821478 podStartE2EDuration="21.253961239s" totalImagesPullingTime="19.149139761s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:38:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:38:36.227844854 +0000 UTC m=+22.675077495" watchObservedRunningTime="2026-09-16 12:38:36.253961239 +0000 UTC m=+22.701193942"
Sep 16 12:38:37.162430 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:37.162114    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6g82p" event={"ID":"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f","Type":"ContainerStarted","Data":"7cd4408bfe98b7e2cbd68267b428b339f41bb71656cc1c7224921b68190af3cc"}
Sep 16 12:38:37.162978 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:37.162457    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6g82p" event={"ID":"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f","Type":"ContainerStarted","Data":"692c6afbde957d31e73f72a97806f51b5b8a9bee1d0a31e88fe38e9b17a89edb"}
Sep 16 12:38:37.162978 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:37.162518    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6g82p" event={"ID":"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f","Type":"ContainerStarted","Data":"931eebcbf3b006841f73a7045a0f18f7aae3325284c8ab9162694236478d6ef7"}
Sep 16 12:38:37.162978 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:37.162566    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6g82p" event={"ID":"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f","Type":"ContainerStarted","Data":"fc16494f5d9fd061c394747477624df677b8ae5bd8565162c9d9b7a01e589513"}
Sep 16 12:38:37.410363 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:37.410323    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 12:38:38.048898 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:38.048864    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmfpk"
Sep 16 12:38:38.049067 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:38.048864    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwb7l"
Sep 16 12:38:38.049067 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:38.048980    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmbgf"
Sep 16 12:38:38.049067 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:38.048972    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmfpk" podUID="aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5"
Sep 16 12:38:38.049176 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:38.049090    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwb7l" podUID="342f831e-0ffe-4582-b616-e0ce14364b32"
Sep 16 12:38:38.049211 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:38.049175    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmbgf" podUID="076f350f-8aef-4c0f-882a-7ac207cf1535"
Sep 16 12:38:38.051453 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:38.051384    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T12:38:37.410345264Z","UUID":"bdfe3456-8bad-4587-81e3-1c31d56ed41c","Handler":null,"Name":"","Endpoint":""}
Sep 16 12:38:38.052936 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:38.052920    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 12:38:38.053047 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:38.052947    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 12:38:38.166580 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:38.166546    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6psq" event={"ID":"a8b1fe11-76c6-4a4a-91a3-865667440c00","Type":"ContainerStarted","Data":"a2d3668a78cfd3d7c5d59f7f9845978e3c9dad59ca5e560197f70f91aa5938db"}
Sep 16 12:38:38.168353 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:38.168323    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-29bsx" event={"ID":"9a4ba4d0-a20b-4ed7-9fc7-92bf3943faf6","Type":"ContainerStarted","Data":"1a3cf4abb66482f8359441762d82ec211c9bb0f0c92eb18acd3d7335b11e28e0"}
Sep 16 12:38:38.197446 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:38.197404    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-29bsx" podStartSLOduration=4.056618236 podStartE2EDuration="23.197388766s" totalImagesPullingTime="19.14077053s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:38:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:38:38.197280296 +0000 UTC m=+24.644512938" watchObservedRunningTime="2026-09-16 12:38:38.197388766 +0000 UTC m=+24.644621409"
Sep 16 12:38:39.174342 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:39.174310    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6g82p" event={"ID":"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f","Type":"ContainerStarted","Data":"3b21d89770e292541a9611876dec8b628ea7a0cb965f4630f9e3ee9d62245a29"}
Sep 16 12:38:39.176347 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:39.176318    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6psq" event={"ID":"a8b1fe11-76c6-4a4a-91a3-865667440c00","Type":"ContainerStarted","Data":"6ad70c0df57996fcd071a8df850d596f786af6b1ff861ccd276af65c7d4452f9"}
Sep 16 12:38:39.197416 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:39.197363    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6psq" podStartSLOduration=2.796179384 podStartE2EDuration="24.197342626s" totalImagesPullingTime="21.401163242s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:38:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:38:39.195555852 +0000 UTC m=+25.642788491" watchObservedRunningTime="2026-09-16 12:38:39.197342626 +0000 UTC m=+25.644575272"
Sep 16 12:38:40.048763 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:40.048726    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmfpk"
Sep 16 12:38:40.048942 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:40.048726    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwb7l"
Sep 16 12:38:40.048942 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:40.048891    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmfpk" podUID="aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5"
Sep 16 12:38:40.049069 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:40.048943    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwb7l" podUID="342f831e-0ffe-4582-b616-e0ce14364b32"
Sep 16 12:38:40.049069 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:40.048742    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmbgf"
Sep 16 12:38:40.049160 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:40.049069    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmbgf" podUID="076f350f-8aef-4c0f-882a-7ac207cf1535"
Sep 16 12:38:41.106429 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:41.106304    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-j2kjn"
Sep 16 12:38:41.106760 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:41.106743    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-j2kjn"
Sep 16 12:38:41.194148 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:41.193913    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6g82p" event={"ID":"e4f4a5d3-fc62-4048-a18f-3ef216a53a9f","Type":"ContainerStarted","Data":"4090e9679fe1609a616f20c02c4c4120a87e89ca00e9d1d1c1bb70672296f36e"}
Sep 16 12:38:41.194148 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:41.194001    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-j2kjn"
Sep 16 12:38:41.194558 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:41.194151    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:41.194558 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:41.194279    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:41.195542 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:41.195240    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-j2kjn"
Sep 16 12:38:41.218303 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:41.215021    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:41.218303 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:41.215227    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:41.224650 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:41.224531    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6g82p" podStartSLOduration=7.065268381 podStartE2EDuration="26.224514455s" totalImagesPullingTime="19.159246074s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:38:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:38:41.222901772 +0000 UTC m=+27.670134415" watchObservedRunningTime="2026-09-16 12:38:41.224514455 +0000 UTC m=+27.671747099"
Sep 16 12:38:42.049006 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:42.048970    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwb7l"
Sep 16 12:38:42.049203 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:42.048970    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmbgf"
Sep 16 12:38:42.049203 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:42.049118    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwb7l" podUID="342f831e-0ffe-4582-b616-e0ce14364b32"
Sep 16 12:38:42.049203 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:42.048972    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmfpk"
Sep 16 12:38:42.049203 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:42.049154    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmbgf" podUID="076f350f-8aef-4c0f-882a-7ac207cf1535"
Sep 16 12:38:42.049419 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:42.049239    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmfpk" podUID="aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5"
Sep 16 12:38:42.196385 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:42.196353    2577 generic.go:356] "Generic (PLEG): container finished" podID="a7e49775-cab4-4ccf-a510-9ed41e681e07" containerID="f7011e9934f46214a30be3f990c6c8dea798b6bd97edd0d9b387d8268adb3a3d" exitCode=0
Sep 16 12:38:42.197110 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:42.196449    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9f6jr" event={"ID":"a7e49775-cab4-4ccf-a510-9ed41e681e07","Type":"ContainerDied","Data":"f7011e9934f46214a30be3f990c6c8dea798b6bd97edd0d9b387d8268adb3a3d"}
Sep 16 12:38:42.197110 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:42.196488    2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 12:38:43.200943 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:43.200742    2577 generic.go:356] "Generic (PLEG): container finished" podID="a7e49775-cab4-4ccf-a510-9ed41e681e07" containerID="69ec2a1c69dd8f42e5790116be59dd3930f6df1b6305215e4806dc4195c86ca7" exitCode=0
Sep 16 12:38:43.201426 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:43.200783    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9f6jr" event={"ID":"a7e49775-cab4-4ccf-a510-9ed41e681e07","Type":"ContainerDied","Data":"69ec2a1c69dd8f42e5790116be59dd3930f6df1b6305215e4806dc4195c86ca7"}
Sep 16 12:38:43.201426 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:43.201095    2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 12:38:43.285574 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:43.285546    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xmbgf"]
Sep 16 12:38:43.285715 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:43.285659    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmbgf"
Sep 16 12:38:43.285754 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:43.285737    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmbgf" podUID="076f350f-8aef-4c0f-882a-7ac207cf1535"
Sep 16 12:38:43.288865 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:43.288837    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tmfpk"]
Sep 16 12:38:43.288996 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:43.288955    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmfpk"
Sep 16 12:38:43.289098 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:43.289077    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmfpk" podUID="aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5"
Sep 16 12:38:43.289451 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:43.289437    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bwb7l"]
Sep 16 12:38:43.289509 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:43.289500    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwb7l"
Sep 16 12:38:43.289571 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:43.289558    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwb7l" podUID="342f831e-0ffe-4582-b616-e0ce14364b32"
Sep 16 12:38:44.205855 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:44.205824    2577 generic.go:356] "Generic (PLEG): container finished" podID="a7e49775-cab4-4ccf-a510-9ed41e681e07" containerID="2f56c338577de392d6588fe62071fd19aba5e7288d27c1ee4768bce9a2e6f147" exitCode=0
Sep 16 12:38:44.206233 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:44.205882    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9f6jr" event={"ID":"a7e49775-cab4-4ccf-a510-9ed41e681e07","Type":"ContainerDied","Data":"2f56c338577de392d6588fe62071fd19aba5e7288d27c1ee4768bce9a2e6f147"}
Sep 16 12:38:45.048698 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:45.048629    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmfpk"
Sep 16 12:38:45.048698 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:45.048641    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwb7l"
Sep 16 12:38:45.048928 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:45.048750    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmbgf"
Sep 16 12:38:45.048928 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:45.048773    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmfpk" podUID="aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5"
Sep 16 12:38:45.048928 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:45.048812    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmbgf" podUID="076f350f-8aef-4c0f-882a-7ac207cf1535"
Sep 16 12:38:45.048928 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:45.048878    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwb7l" podUID="342f831e-0ffe-4582-b616-e0ce14364b32"
Sep 16 12:38:45.447332 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:45.447254    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:38:47.048733 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:47.048676    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwb7l"
Sep 16 12:38:47.048733 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:47.048702    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmfpk"
Sep 16 12:38:47.049205 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:47.048778    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmbgf"
Sep 16 12:38:47.049205 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:47.048799    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwb7l" podUID="342f831e-0ffe-4582-b616-e0ce14364b32"
Sep 16 12:38:47.049205 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:47.048872    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmbgf" podUID="076f350f-8aef-4c0f-882a-7ac207cf1535"
Sep 16 12:38:47.049205 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:47.048964    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmfpk" podUID="aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5"
Sep 16 12:38:47.753171 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:47.753088    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/342f831e-0ffe-4582-b616-e0ce14364b32-original-pull-secret\") pod \"global-pull-secret-syncer-bwb7l\" (UID: \"342f831e-0ffe-4582-b616-e0ce14364b32\") " pod="kube-system/global-pull-secret-syncer-bwb7l"
Sep 16 12:38:47.753171 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:47.753137    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5-metrics-certs\") pod \"network-metrics-daemon-tmfpk\" (UID: \"aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5\") " pod="openshift-multus/network-metrics-daemon-tmfpk"
Sep 16 12:38:47.753409 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:47.753237    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:38:47.753409 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:47.753253    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:38:47.753409 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:47.753305    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/342f831e-0ffe-4582-b616-e0ce14364b32-original-pull-secret podName:342f831e-0ffe-4582-b616-e0ce14364b32 nodeName:}" failed. No retries permitted until 2026-09-16 12:39:19.753282542 +0000 UTC m=+66.200515161 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/342f831e-0ffe-4582-b616-e0ce14364b32-original-pull-secret") pod "global-pull-secret-syncer-bwb7l" (UID: "342f831e-0ffe-4582-b616-e0ce14364b32") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:38:47.753409 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:47.753333    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48bsc\" (UniqueName: \"kubernetes.io/projected/076f350f-8aef-4c0f-882a-7ac207cf1535-kube-api-access-48bsc\") pod \"network-check-target-xmbgf\" (UID: \"076f350f-8aef-4c0f-882a-7ac207cf1535\") " pod="openshift-network-diagnostics/network-check-target-xmbgf"
Sep 16 12:38:47.753409 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:47.753390    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5-metrics-certs podName:aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5 nodeName:}" failed. No retries permitted until 2026-09-16 12:39:19.753377512 +0000 UTC m=+66.200610135 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5-metrics-certs") pod "network-metrics-daemon-tmfpk" (UID: "aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:38:47.753637 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:47.753453    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:38:47.753637 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:47.753476    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:38:47.753637 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:47.753489    2577 projected.go:196] Error preparing data for projected volume kube-api-access-48bsc for pod openshift-network-diagnostics/network-check-target-xmbgf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:38:47.753637 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:47.753542    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/076f350f-8aef-4c0f-882a-7ac207cf1535-kube-api-access-48bsc podName:076f350f-8aef-4c0f-882a-7ac207cf1535 nodeName:}" failed. No retries permitted until 2026-09-16 12:39:19.753527777 +0000 UTC m=+66.200760399 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-48bsc" (UniqueName: "kubernetes.io/projected/076f350f-8aef-4c0f-882a-7ac207cf1535-kube-api-access-48bsc") pod "network-check-target-xmbgf" (UID: "076f350f-8aef-4c0f-882a-7ac207cf1535") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:38:48.897603 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:48.897577    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-190.ec2.internal" event="NodeReady"
Sep 16 12:38:48.897966 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:48.897706    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 12:38:48.947929 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:48.947856    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6fd89fff9d-lkz8m"]
Sep 16 12:38:48.961185 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:48.961158    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m"
Sep 16 12:38:48.964726 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:48.964701    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ct9vp"]
Sep 16 12:38:48.965107 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:48.964896    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 12:38:48.965107 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:48.964896    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 12:38:48.965698 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:48.965266    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 12:38:48.965698 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:48.965544    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5ps7f\""
Sep 16 12:38:48.970404 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:48.970386    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 12:38:48.980828 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:48.980809    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6fd89fff9d-lkz8m"]
Sep 16 12:38:48.980828 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:48.980847    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pjjgz"]
Sep 16 12:38:48.981013 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:48.980936    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ct9vp"
Sep 16 12:38:48.983871 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:48.983854    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 12:38:48.983978 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:48.983960    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 12:38:48.984132 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:48.984114    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fgm9v\""
Sep 16 12:38:48.994896 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:48.994876    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ct9vp"]
Sep 16 12:38:48.994975 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:48.994915    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pjjgz"]
Sep 16 12:38:48.995016 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:48.994987    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pjjgz"
Sep 16 12:38:48.997880 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:48.997861    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 12:38:48.997880 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:48.997871    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 12:38:48.998062 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:48.997947    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-g4k4m\""
Sep 16 12:38:48.998134 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:48.998120    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 12:38:49.048846 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.048818    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmbgf"
Sep 16 12:38:49.048963 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.048818    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmfpk"
Sep 16 12:38:49.049094 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.048818    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwb7l"
Sep 16 12:38:49.051515 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.051491    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 12:38:49.051593 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.051533    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 12:38:49.051593 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.051586    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-b66st\""
Sep 16 12:38:49.051683 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.051669    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-x7vdr\""
Sep 16 12:38:49.051926 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.051911    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 12:38:49.051984 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.051927    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 12:38:49.061917 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.061901    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/70c5a239-65d0-4d57-861b-0ba00290c8bf-image-registry-private-configuration\") pod \"image-registry-6fd89fff9d-lkz8m\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") " pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m"
Sep 16 12:38:49.062062 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.061922    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-bound-sa-token\") pod \"image-registry-6fd89fff9d-lkz8m\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") " pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m"
Sep 16 12:38:49.062062 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.061953    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-registry-tls\") pod \"image-registry-6fd89fff9d-lkz8m\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") " pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m"
Sep 16 12:38:49.062062 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.061969    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7nv6\" (UniqueName: \"kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-kube-api-access-k7nv6\") pod \"image-registry-6fd89fff9d-lkz8m\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") " pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m"
Sep 16 12:38:49.062062 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.062016    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/70c5a239-65d0-4d57-861b-0ba00290c8bf-installation-pull-secrets\") pod \"image-registry-6fd89fff9d-lkz8m\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") " pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m"
Sep 16 12:38:49.062273 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.062067    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4f5ad86c-a72b-4f7e-aa27-484b2e8a3261-config-volume\") pod \"dns-default-ct9vp\" (UID: \"4f5ad86c-a72b-4f7e-aa27-484b2e8a3261\") " pod="openshift-dns/dns-default-ct9vp"
Sep 16 12:38:49.062273 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.062122    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/70c5a239-65d0-4d57-861b-0ba00290c8bf-registry-certificates\") pod \"image-registry-6fd89fff9d-lkz8m\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") " pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m"
Sep 16 12:38:49.062273 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.062142    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmkhq\" (UniqueName: \"kubernetes.io/projected/4f5ad86c-a72b-4f7e-aa27-484b2e8a3261-kube-api-access-lmkhq\") pod \"dns-default-ct9vp\" (UID: \"4f5ad86c-a72b-4f7e-aa27-484b2e8a3261\") " pod="openshift-dns/dns-default-ct9vp"
Sep 16 12:38:49.062273 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.062159    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f5ad86c-a72b-4f7e-aa27-484b2e8a3261-tmp-dir\") pod \"dns-default-ct9vp\" (UID: \"4f5ad86c-a72b-4f7e-aa27-484b2e8a3261\") " pod="openshift-dns/dns-default-ct9vp"
Sep 16 12:38:49.062273 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.062175    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/70c5a239-65d0-4d57-861b-0ba00290c8bf-ca-trust-extracted\") pod \"image-registry-6fd89fff9d-lkz8m\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") " pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m"
Sep 16 12:38:49.062273 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.062198    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/70c5a239-65d0-4d57-861b-0ba00290c8bf-trusted-ca\") pod \"image-registry-6fd89fff9d-lkz8m\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") " pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m"
Sep 16 12:38:49.062273 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.062220    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f5ad86c-a72b-4f7e-aa27-484b2e8a3261-metrics-tls\") pod \"dns-default-ct9vp\" (UID: \"4f5ad86c-a72b-4f7e-aa27-484b2e8a3261\") " pod="openshift-dns/dns-default-ct9vp"
Sep 16 12:38:49.131433 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.131394    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2kjn_c9fcc2dc-c3c3-401c-8534-95cd3908332e/konnectivity-agent/0.log" limit=1024
Sep 16 12:38:49.163340 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.163310    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-registry-tls\") pod \"image-registry-6fd89fff9d-lkz8m\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") " pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m"
Sep 16 12:38:49.163454 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.163346    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7nv6\" (UniqueName: \"kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-kube-api-access-k7nv6\") pod \"image-registry-6fd89fff9d-lkz8m\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") " pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m"
Sep 16 12:38:49.163454 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.163369    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/70c5a239-65d0-4d57-861b-0ba00290c8bf-installation-pull-secrets\") pod \"image-registry-6fd89fff9d-lkz8m\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") " pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m"
Sep 16 12:38:49.163552 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:49.163529    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:38:49.163591 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:49.163555    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fd89fff9d-lkz8m: secret "image-registry-tls" not found
Sep 16 12:38:49.163591 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.163559    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4f5ad86c-a72b-4f7e-aa27-484b2e8a3261-config-volume\") pod \"dns-default-ct9vp\" (UID: \"4f5ad86c-a72b-4f7e-aa27-484b2e8a3261\") " pod="openshift-dns/dns-default-ct9vp"
Sep 16 12:38:49.163697 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:49.163620    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-registry-tls podName:70c5a239-65d0-4d57-861b-0ba00290c8bf nodeName:}" failed. No retries permitted until 2026-09-16 12:38:49.663601619 +0000 UTC m=+36.110834258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-registry-tls") pod "image-registry-6fd89fff9d-lkz8m" (UID: "70c5a239-65d0-4d57-861b-0ba00290c8bf") : secret "image-registry-tls" not found
Sep 16 12:38:49.163697 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.163666    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/70c5a239-65d0-4d57-861b-0ba00290c8bf-registry-certificates\") pod \"image-registry-6fd89fff9d-lkz8m\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") " pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m"
Sep 16 12:38:49.163697 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.163689    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmkhq\" (UniqueName: \"kubernetes.io/projected/4f5ad86c-a72b-4f7e-aa27-484b2e8a3261-kube-api-access-lmkhq\") pod \"dns-default-ct9vp\" (UID: \"4f5ad86c-a72b-4f7e-aa27-484b2e8a3261\") " pod="openshift-dns/dns-default-ct9vp"
Sep 16 12:38:49.163858 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.163707    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6bd5627d-472c-43bf-a550-8f30201dee34-cert\") pod \"ingress-canary-pjjgz\" (UID: \"6bd5627d-472c-43bf-a550-8f30201dee34\") " pod="openshift-ingress-canary/ingress-canary-pjjgz"
Sep 16 12:38:49.163858 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.163722    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n757v\" (UniqueName: \"kubernetes.io/projected/6bd5627d-472c-43bf-a550-8f30201dee34-kube-api-access-n757v\") pod \"ingress-canary-pjjgz\" (UID: \"6bd5627d-472c-43bf-a550-8f30201dee34\") " pod="openshift-ingress-canary/ingress-canary-pjjgz"
Sep 16 12:38:49.163858 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.163752    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f5ad86c-a72b-4f7e-aa27-484b2e8a3261-tmp-dir\") pod \"dns-default-ct9vp\" (UID: \"4f5ad86c-a72b-4f7e-aa27-484b2e8a3261\") " pod="openshift-dns/dns-default-ct9vp"
Sep 16 12:38:49.163858 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.163781    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/70c5a239-65d0-4d57-861b-0ba00290c8bf-ca-trust-extracted\") pod \"image-registry-6fd89fff9d-lkz8m\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") " pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m"
Sep 16 12:38:49.163858 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.163805    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/70c5a239-65d0-4d57-861b-0ba00290c8bf-trusted-ca\") pod \"image-registry-6fd89fff9d-lkz8m\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") " pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m"
Sep 16 12:38:49.163858 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.163830    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f5ad86c-a72b-4f7e-aa27-484b2e8a3261-metrics-tls\") pod \"dns-default-ct9vp\" (UID: \"4f5ad86c-a72b-4f7e-aa27-484b2e8a3261\") " pod="openshift-dns/dns-default-ct9vp"
Sep 16 12:38:49.164201 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:49.163942    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:38:49.164201 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:49.163987    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f5ad86c-a72b-4f7e-aa27-484b2e8a3261-metrics-tls podName:4f5ad86c-a72b-4f7e-aa27-484b2e8a3261 nodeName:}" failed. No retries permitted until 2026-09-16 12:38:49.663972724 +0000 UTC m=+36.111205351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f5ad86c-a72b-4f7e-aa27-484b2e8a3261-metrics-tls") pod "dns-default-ct9vp" (UID: "4f5ad86c-a72b-4f7e-aa27-484b2e8a3261") : secret "dns-default-metrics-tls" not found
Sep 16 12:38:49.164201 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.164014    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/70c5a239-65d0-4d57-861b-0ba00290c8bf-image-registry-private-configuration\") pod \"image-registry-6fd89fff9d-lkz8m\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") " pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m"
Sep 16 12:38:49.164201 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.164067    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-bound-sa-token\") pod \"image-registry-6fd89fff9d-lkz8m\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") " pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m"
Sep 16 12:38:49.164201 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.164071    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f5ad86c-a72b-4f7e-aa27-484b2e8a3261-tmp-dir\") pod \"dns-default-ct9vp\" (UID: \"4f5ad86c-a72b-4f7e-aa27-484b2e8a3261\") " pod="openshift-dns/dns-default-ct9vp"
Sep 16 12:38:49.164461 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.164211    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4f5ad86c-a72b-4f7e-aa27-484b2e8a3261-config-volume\") pod \"dns-default-ct9vp\" (UID: \"4f5ad86c-a72b-4f7e-aa27-484b2e8a3261\") " pod="openshift-dns/dns-default-ct9vp"
Sep 16 12:38:49.164461 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.164294    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/70c5a239-65d0-4d57-861b-0ba00290c8bf-ca-trust-extracted\") pod \"image-registry-6fd89fff9d-lkz8m\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") " pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m"
Sep 16 12:38:49.164461 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.164328    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/70c5a239-65d0-4d57-861b-0ba00290c8bf-registry-certificates\") pod \"image-registry-6fd89fff9d-lkz8m\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") " pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m"
Sep 16 12:38:49.164809 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.164788    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/70c5a239-65d0-4d57-861b-0ba00290c8bf-trusted-ca\") pod \"image-registry-6fd89fff9d-lkz8m\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") " pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m"
Sep 16 12:38:49.167648 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.167632    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/70c5a239-65d0-4d57-861b-0ba00290c8bf-installation-pull-secrets\") pod \"image-registry-6fd89fff9d-lkz8m\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") " pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m"
Sep 16 12:38:49.167706 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.167671    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/70c5a239-65d0-4d57-861b-0ba00290c8bf-image-registry-private-configuration\") pod \"image-registry-6fd89fff9d-lkz8m\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") " pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m"
Sep 16 12:38:49.173432 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.173401    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmkhq\" (UniqueName: \"kubernetes.io/projected/4f5ad86c-a72b-4f7e-aa27-484b2e8a3261-kube-api-access-lmkhq\") pod \"dns-default-ct9vp\" (UID: \"4f5ad86c-a72b-4f7e-aa27-484b2e8a3261\") " pod="openshift-dns/dns-default-ct9vp"
Sep 16 12:38:49.173524 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.173458    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7nv6\" (UniqueName: \"kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-kube-api-access-k7nv6\") pod \"image-registry-6fd89fff9d-lkz8m\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") " pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m"
Sep 16 12:38:49.173775 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.173758    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-bound-sa-token\") pod \"image-registry-6fd89fff9d-lkz8m\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") " pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m"
Sep 16 12:38:49.216105 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.216047    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2kjn_c9fcc2dc-c3c3-401c-8534-95cd3908332e/konnectivity-agent/0.log" limit=1024
Sep 16 12:38:49.264493 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.264469    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6bd5627d-472c-43bf-a550-8f30201dee34-cert\") pod \"ingress-canary-pjjgz\" (UID: \"6bd5627d-472c-43bf-a550-8f30201dee34\") " pod="openshift-ingress-canary/ingress-canary-pjjgz"
Sep 16 12:38:49.264562 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.264509    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n757v\" (UniqueName: \"kubernetes.io/projected/6bd5627d-472c-43bf-a550-8f30201dee34-kube-api-access-n757v\") pod \"ingress-canary-pjjgz\" (UID: \"6bd5627d-472c-43bf-a550-8f30201dee34\") " pod="openshift-ingress-canary/ingress-canary-pjjgz"
Sep 16 12:38:49.267094 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.267071    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6bd5627d-472c-43bf-a550-8f30201dee34-cert\") pod \"ingress-canary-pjjgz\" (UID: \"6bd5627d-472c-43bf-a550-8f30201dee34\") " pod="openshift-ingress-canary/ingress-canary-pjjgz"
Sep 16 12:38:49.273550 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.273524    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n757v\" (UniqueName: \"kubernetes.io/projected/6bd5627d-472c-43bf-a550-8f30201dee34-kube-api-access-n757v\") pod \"ingress-canary-pjjgz\" (UID: \"6bd5627d-472c-43bf-a550-8f30201dee34\") " pod="openshift-ingress-canary/ingress-canary-pjjgz"
Sep 16 12:38:49.303639 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.303621    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pjjgz"
Sep 16 12:38:49.615636 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.615560    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pjjgz"]
Sep 16 12:38:49.618880 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:38:49.618847    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bd5627d_472c_43bf_a550_8f30201dee34.slice/crio-b7aee642fc415ed789e7f0a8291a353d411c6a51368c36469da127a67b8d09d0 WatchSource:0}: Error finding container b7aee642fc415ed789e7f0a8291a353d411c6a51368c36469da127a67b8d09d0: Status 404 returned error can't find the container with id b7aee642fc415ed789e7f0a8291a353d411c6a51368c36469da127a67b8d09d0
Sep 16 12:38:49.668108 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.668085    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f5ad86c-a72b-4f7e-aa27-484b2e8a3261-metrics-tls\") pod \"dns-default-ct9vp\" (UID: \"4f5ad86c-a72b-4f7e-aa27-484b2e8a3261\") " pod="openshift-dns/dns-default-ct9vp"
Sep 16 12:38:49.668179 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:49.668145    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-registry-tls\") pod \"image-registry-6fd89fff9d-lkz8m\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") " pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m"
Sep 16 12:38:49.668257 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:49.668239    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:38:49.668297 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:49.668262    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fd89fff9d-lkz8m: secret "image-registry-tls" not found
Sep 16 12:38:49.668297 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:49.668243    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:38:49.668353 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:49.668311    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-registry-tls podName:70c5a239-65d0-4d57-861b-0ba00290c8bf nodeName:}" failed. No retries permitted until 2026-09-16 12:38:50.668297489 +0000 UTC m=+37.115530109 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-registry-tls") pod "image-registry-6fd89fff9d-lkz8m" (UID: "70c5a239-65d0-4d57-861b-0ba00290c8bf") : secret "image-registry-tls" not found
Sep 16 12:38:49.668353 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:49.668324    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f5ad86c-a72b-4f7e-aa27-484b2e8a3261-metrics-tls podName:4f5ad86c-a72b-4f7e-aa27-484b2e8a3261 nodeName:}" failed. No retries permitted until 2026-09-16 12:38:50.668318271 +0000 UTC m=+37.115550891 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f5ad86c-a72b-4f7e-aa27-484b2e8a3261-metrics-tls") pod "dns-default-ct9vp" (UID: "4f5ad86c-a72b-4f7e-aa27-484b2e8a3261") : secret "dns-default-metrics-tls" not found
Sep 16 12:38:50.223168 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:50.223133    2577 generic.go:356] "Generic (PLEG): container finished" podID="a7e49775-cab4-4ccf-a510-9ed41e681e07" containerID="c325237b4da35bf79284eaf8d24f1096c6afbe66757242817eebf457b1d840c8" exitCode=0
Sep 16 12:38:50.224097 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:50.223226    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9f6jr" event={"ID":"a7e49775-cab4-4ccf-a510-9ed41e681e07","Type":"ContainerDied","Data":"c325237b4da35bf79284eaf8d24f1096c6afbe66757242817eebf457b1d840c8"}
Sep 16 12:38:50.224461 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:50.224428    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pjjgz" event={"ID":"6bd5627d-472c-43bf-a550-8f30201dee34","Type":"ContainerStarted","Data":"b7aee642fc415ed789e7f0a8291a353d411c6a51368c36469da127a67b8d09d0"}
Sep 16 12:38:50.674834 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:50.674799    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f5ad86c-a72b-4f7e-aa27-484b2e8a3261-metrics-tls\") pod \"dns-default-ct9vp\" (UID: \"4f5ad86c-a72b-4f7e-aa27-484b2e8a3261\") " pod="openshift-dns/dns-default-ct9vp"
Sep 16 12:38:50.674979 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:50.674878    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-registry-tls\") pod \"image-registry-6fd89fff9d-lkz8m\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") " pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m"
Sep 16 12:38:50.674979 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:50.674950    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:38:50.675088 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:50.674986    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:38:50.675088 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:50.674997    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fd89fff9d-lkz8m: secret "image-registry-tls" not found
Sep 16 12:38:50.675088 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:50.675041    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f5ad86c-a72b-4f7e-aa27-484b2e8a3261-metrics-tls podName:4f5ad86c-a72b-4f7e-aa27-484b2e8a3261 nodeName:}" failed. No retries permitted until 2026-09-16 12:38:52.675009994 +0000 UTC m=+39.122242620 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f5ad86c-a72b-4f7e-aa27-484b2e8a3261-metrics-tls") pod "dns-default-ct9vp" (UID: "4f5ad86c-a72b-4f7e-aa27-484b2e8a3261") : secret "dns-default-metrics-tls" not found
Sep 16 12:38:50.675088 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:50.675057    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-registry-tls podName:70c5a239-65d0-4d57-861b-0ba00290c8bf nodeName:}" failed. No retries permitted until 2026-09-16 12:38:52.675050526 +0000 UTC m=+39.122283146 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-registry-tls") pod "image-registry-6fd89fff9d-lkz8m" (UID: "70c5a239-65d0-4d57-861b-0ba00290c8bf") : secret "image-registry-tls" not found
Sep 16 12:38:51.231532 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:51.231327    2577 generic.go:356] "Generic (PLEG): container finished" podID="a7e49775-cab4-4ccf-a510-9ed41e681e07" containerID="b42880f7df89a35605f7a9f6d1a30ca19e11edf60feec9aa7dfef343d0f76268" exitCode=0
Sep 16 12:38:51.231961 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:51.231399    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9f6jr" event={"ID":"a7e49775-cab4-4ccf-a510-9ed41e681e07","Type":"ContainerDied","Data":"b42880f7df89a35605f7a9f6d1a30ca19e11edf60feec9aa7dfef343d0f76268"}
Sep 16 12:38:52.237935 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:52.237897    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9f6jr" event={"ID":"a7e49775-cab4-4ccf-a510-9ed41e681e07","Type":"ContainerStarted","Data":"acd89ed3096e3a2c38a91e2702d811f9eef0767b4acc2c4bbf7cb476d7b23ff7"}
Sep 16 12:38:52.239555 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:52.239480    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pjjgz" event={"ID":"6bd5627d-472c-43bf-a550-8f30201dee34","Type":"ContainerStarted","Data":"c0f97ce636046699955599e44364688865aa3f6234bad144f3318890c52d04b5"}
Sep 16 12:38:52.267172 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:52.267119    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9f6jr" podStartSLOduration=6.503949023 podStartE2EDuration="37.267103231s" totalImagesPullingTime="30.763154208s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:38:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:38:52.265443794 +0000 UTC m=+38.712676435" watchObservedRunningTime="2026-09-16 12:38:52.267103231 +0000 UTC m=+38.714335874"
Sep 16 12:38:52.288413 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:52.288371    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pjjgz" podStartSLOduration=2.011282738 podStartE2EDuration="4.288357291s" totalImagesPullingTime="2.277074553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:38:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:38:52.286183232 +0000 UTC m=+38.733415873" watchObservedRunningTime="2026-09-16 12:38:52.288357291 +0000 UTC m=+38.735589937"
Sep 16 12:38:52.417648 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:52.417608    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2kjn_c9fcc2dc-c3c3-401c-8534-95cd3908332e/konnectivity-agent/0.log" limit=1024
Sep 16 12:38:52.512758 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:52.512674    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2kjn_c9fcc2dc-c3c3-401c-8534-95cd3908332e/konnectivity-agent/0.log" limit=1024
Sep 16 12:38:52.614657 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:52.614625    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2kjn_c9fcc2dc-c3c3-401c-8534-95cd3908332e/konnectivity-agent/0.log" limit=1024
Sep 16 12:38:52.689095 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:52.689071    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f5ad86c-a72b-4f7e-aa27-484b2e8a3261-metrics-tls\") pod \"dns-default-ct9vp\" (UID: \"4f5ad86c-a72b-4f7e-aa27-484b2e8a3261\") " pod="openshift-dns/dns-default-ct9vp"
Sep 16 12:38:52.689174 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:52.689118    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-registry-tls\") pod \"image-registry-6fd89fff9d-lkz8m\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") " pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m"
Sep 16 12:38:52.689233 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:52.689220    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:38:52.689274 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:52.689223    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:38:52.689274 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:52.689233    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fd89fff9d-lkz8m: secret "image-registry-tls" not found
Sep 16 12:38:52.689343 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:52.689285    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-registry-tls podName:70c5a239-65d0-4d57-861b-0ba00290c8bf nodeName:}" failed. No retries permitted until 2026-09-16 12:38:56.689270982 +0000 UTC m=+43.136503601 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-registry-tls") pod "image-registry-6fd89fff9d-lkz8m" (UID: "70c5a239-65d0-4d57-861b-0ba00290c8bf") : secret "image-registry-tls" not found
Sep 16 12:38:52.689343 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:52.689318    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f5ad86c-a72b-4f7e-aa27-484b2e8a3261-metrics-tls podName:4f5ad86c-a72b-4f7e-aa27-484b2e8a3261 nodeName:}" failed. No retries permitted until 2026-09-16 12:38:56.689304187 +0000 UTC m=+43.136536807 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f5ad86c-a72b-4f7e-aa27-484b2e8a3261-metrics-tls") pod "dns-default-ct9vp" (UID: "4f5ad86c-a72b-4f7e-aa27-484b2e8a3261") : secret "dns-default-metrics-tls" not found
Sep 16 12:38:53.006363 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:53.006328    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2kjn_c9fcc2dc-c3c3-401c-8534-95cd3908332e/konnectivity-agent/0.log" limit=1024
Sep 16 12:38:53.107052 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:53.107002    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2kjn_c9fcc2dc-c3c3-401c-8534-95cd3908332e/konnectivity-agent/0.log" limit=1024
Sep 16 12:38:54.182071 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:54.182040    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2kjn_c9fcc2dc-c3c3-401c-8534-95cd3908332e/konnectivity-agent/0.log" limit=1024
Sep 16 12:38:54.244919 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:54.244885    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2kjn_c9fcc2dc-c3c3-401c-8534-95cd3908332e/konnectivity-agent/0.log" limit=1024
Sep 16 12:38:56.718637 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:56.718595    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f5ad86c-a72b-4f7e-aa27-484b2e8a3261-metrics-tls\") pod \"dns-default-ct9vp\" (UID: \"4f5ad86c-a72b-4f7e-aa27-484b2e8a3261\") " pod="openshift-dns/dns-default-ct9vp"
Sep 16 12:38:56.719061 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:56.718673    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-registry-tls\") pod \"image-registry-6fd89fff9d-lkz8m\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") " pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m"
Sep 16 12:38:56.719061 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:56.718759    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:38:56.719061 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:56.718824    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f5ad86c-a72b-4f7e-aa27-484b2e8a3261-metrics-tls podName:4f5ad86c-a72b-4f7e-aa27-484b2e8a3261 nodeName:}" failed. No retries permitted until 2026-09-16 12:39:04.718808814 +0000 UTC m=+51.166041447 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f5ad86c-a72b-4f7e-aa27-484b2e8a3261-metrics-tls") pod "dns-default-ct9vp" (UID: "4f5ad86c-a72b-4f7e-aa27-484b2e8a3261") : secret "dns-default-metrics-tls" not found
Sep 16 12:38:56.719061 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:56.718762    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:38:56.719061 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:56.718865    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fd89fff9d-lkz8m: secret "image-registry-tls" not found
Sep 16 12:38:56.719061 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:38:56.718911    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-registry-tls podName:70c5a239-65d0-4d57-861b-0ba00290c8bf nodeName:}" failed. No retries permitted until 2026-09-16 12:39:04.718897184 +0000 UTC m=+51.166129804 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-registry-tls") pod "image-registry-6fd89fff9d-lkz8m" (UID: "70c5a239-65d0-4d57-861b-0ba00290c8bf") : secret "image-registry-tls" not found
Sep 16 12:38:56.872732 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:56.872693    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2kjn_c9fcc2dc-c3c3-401c-8534-95cd3908332e/konnectivity-agent/0.log" limit=1024
Sep 16 12:38:56.976477 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:56.976397    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2kjn_c9fcc2dc-c3c3-401c-8534-95cd3908332e/konnectivity-agent/0.log" limit=1024
Sep 16 12:38:57.859957 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:57.859925    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2kjn_c9fcc2dc-c3c3-401c-8534-95cd3908332e/konnectivity-agent/0.log" limit=1024
Sep 16 12:38:57.917564 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:38:57.917536    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2kjn_c9fcc2dc-c3c3-401c-8534-95cd3908332e/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:04.776295 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:39:04.776252    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f5ad86c-a72b-4f7e-aa27-484b2e8a3261-metrics-tls\") pod \"dns-default-ct9vp\" (UID: \"4f5ad86c-a72b-4f7e-aa27-484b2e8a3261\") " pod="openshift-dns/dns-default-ct9vp"
Sep 16 12:39:04.776766 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:39:04.776323    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-registry-tls\") pod \"image-registry-6fd89fff9d-lkz8m\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") " pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m"
Sep 16 12:39:04.776766 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:39:04.776398    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:39:04.776766 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:39:04.776453    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:39:04.776766 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:39:04.776473    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fd89fff9d-lkz8m: secret "image-registry-tls" not found
Sep 16 12:39:04.776766 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:39:04.776463    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f5ad86c-a72b-4f7e-aa27-484b2e8a3261-metrics-tls podName:4f5ad86c-a72b-4f7e-aa27-484b2e8a3261 nodeName:}" failed. No retries permitted until 2026-09-16 12:39:20.77644485 +0000 UTC m=+67.223677487 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f5ad86c-a72b-4f7e-aa27-484b2e8a3261-metrics-tls") pod "dns-default-ct9vp" (UID: "4f5ad86c-a72b-4f7e-aa27-484b2e8a3261") : secret "dns-default-metrics-tls" not found
Sep 16 12:39:04.776766 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:39:04.776529    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-registry-tls podName:70c5a239-65d0-4d57-861b-0ba00290c8bf nodeName:}" failed. No retries permitted until 2026-09-16 12:39:20.776517329 +0000 UTC m=+67.223749949 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-registry-tls") pod "image-registry-6fd89fff9d-lkz8m" (UID: "70c5a239-65d0-4d57-861b-0ba00290c8bf") : secret "image-registry-tls" not found
Sep 16 12:39:15.457771 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:39:15.457740    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6g82p"
Sep 16 12:39:19.778255 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:39:19.778217    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5-metrics-certs\") pod \"network-metrics-daemon-tmfpk\" (UID: \"aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5\") " pod="openshift-multus/network-metrics-daemon-tmfpk"
Sep 16 12:39:19.778647 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:39:19.778275    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48bsc\" (UniqueName: \"kubernetes.io/projected/076f350f-8aef-4c0f-882a-7ac207cf1535-kube-api-access-48bsc\") pod \"network-check-target-xmbgf\" (UID: \"076f350f-8aef-4c0f-882a-7ac207cf1535\") " pod="openshift-network-diagnostics/network-check-target-xmbgf"
Sep 16 12:39:19.778647 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:39:19.778324    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/342f831e-0ffe-4582-b616-e0ce14364b32-original-pull-secret\") pod \"global-pull-secret-syncer-bwb7l\" (UID: \"342f831e-0ffe-4582-b616-e0ce14364b32\") " pod="kube-system/global-pull-secret-syncer-bwb7l"
Sep 16 12:39:19.781608 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:39:19.781585    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 12:39:19.781608 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:39:19.781595    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 12:39:19.781778 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:39:19.781647    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 12:39:19.788834 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:39:19.788815    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 12:39:19.788913 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:39:19.788884    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5-metrics-certs podName:aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:23.788869301 +0000 UTC m=+130.236101920 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5-metrics-certs") pod "network-metrics-daemon-tmfpk" (UID: "aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5") : secret "metrics-daemon-secret" not found
Sep 16 12:39:19.791241 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:39:19.791226    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 12:39:19.792182 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:39:19.792165    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/342f831e-0ffe-4582-b616-e0ce14364b32-original-pull-secret\") pod \"global-pull-secret-syncer-bwb7l\" (UID: \"342f831e-0ffe-4582-b616-e0ce14364b32\") " pod="kube-system/global-pull-secret-syncer-bwb7l"
Sep 16 12:39:19.801596 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:39:19.801578    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48bsc\" (UniqueName: \"kubernetes.io/projected/076f350f-8aef-4c0f-882a-7ac207cf1535-kube-api-access-48bsc\") pod \"network-check-target-xmbgf\" (UID: \"076f350f-8aef-4c0f-882a-7ac207cf1535\") " pod="openshift-network-diagnostics/network-check-target-xmbgf"
Sep 16 12:39:19.961331 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:39:19.961302    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-x7vdr\""
Sep 16 12:39:19.968535 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:39:19.968513    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmbgf"
Sep 16 12:39:19.968609 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:39:19.968533    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwb7l"
Sep 16 12:39:20.267250 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:39:20.267139    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xmbgf"]
Sep 16 12:39:20.269777 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:39:20.269753    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod076f350f_8aef_4c0f_882a_7ac207cf1535.slice/crio-f22daf1889fb1c34f74f59c1482fa253458db9dc5b7fe39e13a212a0cd5815a0 WatchSource:0}: Error finding container f22daf1889fb1c34f74f59c1482fa253458db9dc5b7fe39e13a212a0cd5815a0: Status 404 returned error can't find the container with id f22daf1889fb1c34f74f59c1482fa253458db9dc5b7fe39e13a212a0cd5815a0
Sep 16 12:39:20.291798 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:39:20.291746    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bwb7l"]
Sep 16 12:39:20.295080 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:39:20.295060    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod342f831e_0ffe_4582_b616_e0ce14364b32.slice/crio-8cc86d870c30004cbb9f16eb3b7f75ffd61b7738f47eb1cff7ee2b00549c4832 WatchSource:0}: Error finding container 8cc86d870c30004cbb9f16eb3b7f75ffd61b7738f47eb1cff7ee2b00549c4832: Status 404 returned error can't find the container with id 8cc86d870c30004cbb9f16eb3b7f75ffd61b7738f47eb1cff7ee2b00549c4832
Sep 16 12:39:20.305631 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:39:20.305606    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bwb7l" event={"ID":"342f831e-0ffe-4582-b616-e0ce14364b32","Type":"ContainerStarted","Data":"8cc86d870c30004cbb9f16eb3b7f75ffd61b7738f47eb1cff7ee2b00549c4832"}
Sep 16 12:39:20.306633 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:39:20.306613    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xmbgf" event={"ID":"076f350f-8aef-4c0f-882a-7ac207cf1535","Type":"ContainerStarted","Data":"f22daf1889fb1c34f74f59c1482fa253458db9dc5b7fe39e13a212a0cd5815a0"}
Sep 16 12:39:20.785744 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:39:20.785712    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-registry-tls\") pod \"image-registry-6fd89fff9d-lkz8m\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") " pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m"
Sep 16 12:39:20.786205 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:39:20.785780    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f5ad86c-a72b-4f7e-aa27-484b2e8a3261-metrics-tls\") pod \"dns-default-ct9vp\" (UID: \"4f5ad86c-a72b-4f7e-aa27-484b2e8a3261\") " pod="openshift-dns/dns-default-ct9vp"
Sep 16 12:39:20.786205 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:39:20.785880    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:39:20.786205 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:39:20.785899    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fd89fff9d-lkz8m: secret "image-registry-tls" not found
Sep 16 12:39:20.786205 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:39:20.785909    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:39:20.786205 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:39:20.785968    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f5ad86c-a72b-4f7e-aa27-484b2e8a3261-metrics-tls podName:4f5ad86c-a72b-4f7e-aa27-484b2e8a3261 nodeName:}" failed. No retries permitted until 2026-09-16 12:39:52.785949915 +0000 UTC m=+99.233182552 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f5ad86c-a72b-4f7e-aa27-484b2e8a3261-metrics-tls") pod "dns-default-ct9vp" (UID: "4f5ad86c-a72b-4f7e-aa27-484b2e8a3261") : secret "dns-default-metrics-tls" not found
Sep 16 12:39:20.786205 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:39:20.785989    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-registry-tls podName:70c5a239-65d0-4d57-861b-0ba00290c8bf nodeName:}" failed. No retries permitted until 2026-09-16 12:39:52.785978595 +0000 UTC m=+99.233211219 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-registry-tls") pod "image-registry-6fd89fff9d-lkz8m" (UID: "70c5a239-65d0-4d57-861b-0ba00290c8bf") : secret "image-registry-tls" not found
Sep 16 12:39:25.321236 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:39:25.321193    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bwb7l" event={"ID":"342f831e-0ffe-4582-b616-e0ce14364b32","Type":"ContainerStarted","Data":"b859dfb97314d7c56f44d65d14629264b22801086a8c7545223ff43502db67d8"}
Sep 16 12:39:25.322794 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:39:25.322771    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xmbgf" event={"ID":"076f350f-8aef-4c0f-882a-7ac207cf1535","Type":"ContainerStarted","Data":"06d0d310196b695abc495c5909ce2f6e824e3d26859037fab57d9cc3c5b58043"}
Sep 16 12:39:25.322928 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:39:25.322909    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xmbgf"
Sep 16 12:39:25.338070 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:39:25.338011    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-bwb7l" podStartSLOduration=65.720257974 podStartE2EDuration="1m10.337998117s" totalImagesPullingTime="4.617740143s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:38:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:39:25.337704148 +0000 UTC m=+71.784936803" watchObservedRunningTime="2026-09-16 12:39:25.337998117 +0000 UTC m=+71.785230741"
Sep 16 12:39:25.358875 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:39:25.358822    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xmbgf" podStartSLOduration=65.723572146 podStartE2EDuration="1m10.358807827s" totalImagesPullingTime="4.635235681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:38:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:39:25.358363998 +0000 UTC m=+71.805596663" watchObservedRunningTime="2026-09-16 12:39:25.358807827 +0000 UTC m=+71.806040469"
Sep 16 12:39:52.807267 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:39:52.807224    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-registry-tls\") pod \"image-registry-6fd89fff9d-lkz8m\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") " pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m"
Sep 16 12:39:52.807651 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:39:52.807294    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f5ad86c-a72b-4f7e-aa27-484b2e8a3261-metrics-tls\") pod \"dns-default-ct9vp\" (UID: \"4f5ad86c-a72b-4f7e-aa27-484b2e8a3261\") " pod="openshift-dns/dns-default-ct9vp"
Sep 16 12:39:52.807651 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:39:52.807382    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:39:52.807651 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:39:52.807387    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:39:52.807651 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:39:52.807408    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fd89fff9d-lkz8m: secret "image-registry-tls" not found
Sep 16 12:39:52.807651 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:39:52.807433    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f5ad86c-a72b-4f7e-aa27-484b2e8a3261-metrics-tls podName:4f5ad86c-a72b-4f7e-aa27-484b2e8a3261 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:56.807420062 +0000 UTC m=+163.254652683 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f5ad86c-a72b-4f7e-aa27-484b2e8a3261-metrics-tls") pod "dns-default-ct9vp" (UID: "4f5ad86c-a72b-4f7e-aa27-484b2e8a3261") : secret "dns-default-metrics-tls" not found
Sep 16 12:39:52.807651 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:39:52.807458    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-registry-tls podName:70c5a239-65d0-4d57-861b-0ba00290c8bf nodeName:}" failed. No retries permitted until 2026-09-16 12:40:56.807444773 +0000 UTC m=+163.254677393 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-registry-tls") pod "image-registry-6fd89fff9d-lkz8m" (UID: "70c5a239-65d0-4d57-861b-0ba00290c8bf") : secret "image-registry-tls" not found
Sep 16 12:39:56.328615 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:39:56.328584    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xmbgf"
Sep 16 12:40:23.818415 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:23.818379    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5-metrics-certs\") pod \"network-metrics-daemon-tmfpk\" (UID: \"aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5\") " pod="openshift-multus/network-metrics-daemon-tmfpk"
Sep 16 12:40:23.818893 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:23.818501    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 12:40:23.818893 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:23.818563    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5-metrics-certs podName:aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5 nodeName:}" failed. No retries permitted until 2026-09-16 12:42:25.818548981 +0000 UTC m=+252.265781602 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5-metrics-certs") pod "network-metrics-daemon-tmfpk" (UID: "aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5") : secret "metrics-daemon-secret" not found
Sep 16 12:40:36.068867 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.068834    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-mlzkb"]
Sep 16 12:40:36.071478 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.071462    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mlzkb"
Sep 16 12:40:36.077973 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.077464    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 16 12:40:36.080709 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.078666    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 12:40:36.080709 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.079002    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 12:40:36.081239 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.081214    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 16 12:40:36.081369 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.081220    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-xrnr8\""
Sep 16 12:40:36.082989 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.082969    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-mlzkb"]
Sep 16 12:40:36.172130 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.172103    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-qfll2"]
Sep 16 12:40:36.174856 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.174842    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-qfll2"
Sep 16 12:40:36.178890 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.178869    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-l6mdz\""
Sep 16 12:40:36.178998 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.178912    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5q4bl"]
Sep 16 12:40:36.181648 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.181631    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-mhv8x"]
Sep 16 12:40:36.181763 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.181751    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5q4bl"
Sep 16 12:40:36.184362 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.184342    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 16 12:40:36.184449 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.184427    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-vbt8d"]
Sep 16 12:40:36.184575 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.184559    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-mhv8x"
Sep 16 12:40:36.184724 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.184708    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-dxrdw\""
Sep 16 12:40:36.184959 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.184939    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:40:36.185099 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.184969    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 16 12:40:36.185099 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.184942    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 16 12:40:36.187199 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.187183    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-vbt8d"
Sep 16 12:40:36.187824 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.187802    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-4k25q\""
Sep 16 12:40:36.187931 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.187918    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 16 12:40:36.188528 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.188507    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:40:36.191064 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.191020    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:40:36.191435 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.191416    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-qfll2"]
Sep 16 12:40:36.191684 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.191667    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 16 12:40:36.191944 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.191922    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 16 12:40:36.192072 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.191968    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-tt8x6\""
Sep 16 12:40:36.192072 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.191982    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 16 12:40:36.197658 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.197638    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-mhv8x"]
Sep 16 12:40:36.198260 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.198244    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 16 12:40:36.198465 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.198450    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5q4bl"]
Sep 16 12:40:36.202608 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.202586    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgkdp\" (UniqueName: \"kubernetes.io/projected/a583c3d2-5168-4901-89ec-2c64993e60ec-kube-api-access-vgkdp\") pod \"cluster-monitoring-operator-6cd954b897-mlzkb\" (UID: \"a583c3d2-5168-4901-89ec-2c64993e60ec\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mlzkb"
Sep 16 12:40:36.202768 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.202634    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a583c3d2-5168-4901-89ec-2c64993e60ec-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-mlzkb\" (UID: \"a583c3d2-5168-4901-89ec-2c64993e60ec\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mlzkb"
Sep 16 12:40:36.202768 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.202688    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a583c3d2-5168-4901-89ec-2c64993e60ec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-mlzkb\" (UID: \"a583c3d2-5168-4901-89ec-2c64993e60ec\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mlzkb"
Sep 16 12:40:36.203445 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.203423    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-vbt8d"]
Sep 16 12:40:36.276804 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.276773    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-944d5c59d-l5g2b"]
Sep 16 12:40:36.279686 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.279672    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-944d5c59d-l5g2b"
Sep 16 12:40:36.282637 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.282616    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 16 12:40:36.283087 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.283068    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 16 12:40:36.283183 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.283131    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 12:40:36.284930 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.283856    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 16 12:40:36.284930 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.283906    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 16 12:40:36.284930 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.283921    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xhxrf\""
Sep 16 12:40:36.288183 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.286985    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 12:40:36.298271 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.298253    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-944d5c59d-l5g2b"]
Sep 16 12:40:36.303634 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.303615    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ad668dd0-69c8-435f-9b75-32f037bfdbc1-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5q4bl\" (UID: \"ad668dd0-69c8-435f-9b75-32f037bfdbc1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5q4bl"
Sep 16 12:40:36.303733 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.303642    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ad668dd0-69c8-435f-9b75-32f037bfdbc1-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5q4bl\" (UID: \"ad668dd0-69c8-435f-9b75-32f037bfdbc1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5q4bl"
Sep 16 12:40:36.303733 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.303659    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6d6b\" (UniqueName: \"kubernetes.io/projected/ad668dd0-69c8-435f-9b75-32f037bfdbc1-kube-api-access-q6d6b\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5q4bl\" (UID: \"ad668dd0-69c8-435f-9b75-32f037bfdbc1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5q4bl"
Sep 16 12:40:36.303733 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.303684    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgkdp\" (UniqueName: \"kubernetes.io/projected/a583c3d2-5168-4901-89ec-2c64993e60ec-kube-api-access-vgkdp\") pod \"cluster-monitoring-operator-6cd954b897-mlzkb\" (UID: \"a583c3d2-5168-4901-89ec-2c64993e60ec\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mlzkb"
Sep 16 12:40:36.303733 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.303700    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljbg4\" (UniqueName: \"kubernetes.io/projected/e0992b2d-fbf2-4e89-afbc-543ce47e0615-kube-api-access-ljbg4\") pod \"volume-data-source-validator-86695b7cc7-mhv8x\" (UID: \"e0992b2d-fbf2-4e89-afbc-543ce47e0615\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-mhv8x"
Sep 16 12:40:36.303931 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.303788    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a583c3d2-5168-4901-89ec-2c64993e60ec-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-mlzkb\" (UID: \"a583c3d2-5168-4901-89ec-2c64993e60ec\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mlzkb"
Sep 16 12:40:36.303931 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.303820    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174-trusted-ca\") pod \"console-operator-696884fb9c-vbt8d\" (UID: \"0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174\") " pod="openshift-console-operator/console-operator-696884fb9c-vbt8d"
Sep 16 12:40:36.303931 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.303837    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq699\" (UniqueName: \"kubernetes.io/projected/9306d26d-8a34-4f8d-91da-68ba20f0afa0-kube-api-access-mq699\") pod \"network-check-source-6f8b768455-qfll2\" (UID: \"9306d26d-8a34-4f8d-91da-68ba20f0afa0\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-qfll2"
Sep 16 12:40:36.303931 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.303867    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x2q9\" (UniqueName: \"kubernetes.io/projected/0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174-kube-api-access-8x2q9\") pod \"console-operator-696884fb9c-vbt8d\" (UID: \"0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174\") " pod="openshift-console-operator/console-operator-696884fb9c-vbt8d"
Sep 16 12:40:36.303931 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.303894    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174-config\") pod \"console-operator-696884fb9c-vbt8d\" (UID: \"0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174\") " pod="openshift-console-operator/console-operator-696884fb9c-vbt8d"
Sep 16 12:40:36.303931 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.303914    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174-serving-cert\") pod \"console-operator-696884fb9c-vbt8d\" (UID: \"0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174\") " pod="openshift-console-operator/console-operator-696884fb9c-vbt8d"
Sep 16 12:40:36.304269 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.303937    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a583c3d2-5168-4901-89ec-2c64993e60ec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-mlzkb\" (UID: \"a583c3d2-5168-4901-89ec-2c64993e60ec\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mlzkb"
Sep 16 12:40:36.304269 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:36.304011    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 12:40:36.304269 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:36.304090    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a583c3d2-5168-4901-89ec-2c64993e60ec-cluster-monitoring-operator-tls podName:a583c3d2-5168-4901-89ec-2c64993e60ec nodeName:}" failed. No retries permitted until 2026-09-16 12:40:36.804074593 +0000 UTC m=+143.251307220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a583c3d2-5168-4901-89ec-2c64993e60ec-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-mlzkb" (UID: "a583c3d2-5168-4901-89ec-2c64993e60ec") : secret "cluster-monitoring-operator-tls" not found
Sep 16 12:40:36.304532 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.304513    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a583c3d2-5168-4901-89ec-2c64993e60ec-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-mlzkb\" (UID: \"a583c3d2-5168-4901-89ec-2c64993e60ec\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mlzkb"
Sep 16 12:40:36.313483 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.313463    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgkdp\" (UniqueName: \"kubernetes.io/projected/a583c3d2-5168-4901-89ec-2c64993e60ec-kube-api-access-vgkdp\") pod \"cluster-monitoring-operator-6cd954b897-mlzkb\" (UID: \"a583c3d2-5168-4901-89ec-2c64993e60ec\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mlzkb"
Sep 16 12:40:36.405166 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.405089    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b04ec1dc-5e71-4fed-abc5-6875befad478-default-certificate\") pod \"router-default-944d5c59d-l5g2b\" (UID: \"b04ec1dc-5e71-4fed-abc5-6875befad478\") " pod="openshift-ingress/router-default-944d5c59d-l5g2b"
Sep 16 12:40:36.405166 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.405122    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b04ec1dc-5e71-4fed-abc5-6875befad478-stats-auth\") pod \"router-default-944d5c59d-l5g2b\" (UID: \"b04ec1dc-5e71-4fed-abc5-6875befad478\") " pod="openshift-ingress/router-default-944d5c59d-l5g2b"
Sep 16 12:40:36.405166 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.405159    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ad668dd0-69c8-435f-9b75-32f037bfdbc1-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5q4bl\" (UID: \"ad668dd0-69c8-435f-9b75-32f037bfdbc1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5q4bl"
Sep 16 12:40:36.405425 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.405203    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljbg4\" (UniqueName: \"kubernetes.io/projected/e0992b2d-fbf2-4e89-afbc-543ce47e0615-kube-api-access-ljbg4\") pod \"volume-data-source-validator-86695b7cc7-mhv8x\" (UID: \"e0992b2d-fbf2-4e89-afbc-543ce47e0615\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-mhv8x"
Sep 16 12:40:36.405425 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.405255    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq699\" (UniqueName: \"kubernetes.io/projected/9306d26d-8a34-4f8d-91da-68ba20f0afa0-kube-api-access-mq699\") pod \"network-check-source-6f8b768455-qfll2\" (UID: \"9306d26d-8a34-4f8d-91da-68ba20f0afa0\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-qfll2"
Sep 16 12:40:36.405425 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.405282    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6d6b\" (UniqueName: \"kubernetes.io/projected/ad668dd0-69c8-435f-9b75-32f037bfdbc1-kube-api-access-q6d6b\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5q4bl\" (UID: \"ad668dd0-69c8-435f-9b75-32f037bfdbc1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5q4bl"
Sep 16 12:40:36.405425 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.405308    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x2q9\" (UniqueName: \"kubernetes.io/projected/0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174-kube-api-access-8x2q9\") pod \"console-operator-696884fb9c-vbt8d\" (UID: \"0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174\") " pod="openshift-console-operator/console-operator-696884fb9c-vbt8d"
Sep 16 12:40:36.405425 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.405363    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd4bs\" (UniqueName: \"kubernetes.io/projected/b04ec1dc-5e71-4fed-abc5-6875befad478-kube-api-access-hd4bs\") pod \"router-default-944d5c59d-l5g2b\" (UID: \"b04ec1dc-5e71-4fed-abc5-6875befad478\") " pod="openshift-ingress/router-default-944d5c59d-l5g2b"
Sep 16 12:40:36.405425 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.405404    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b04ec1dc-5e71-4fed-abc5-6875befad478-metrics-certs\") pod \"router-default-944d5c59d-l5g2b\" (UID: \"b04ec1dc-5e71-4fed-abc5-6875befad478\") " pod="openshift-ingress/router-default-944d5c59d-l5g2b"
Sep 16 12:40:36.405677 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.405431    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b04ec1dc-5e71-4fed-abc5-6875befad478-service-ca-bundle\") pod \"router-default-944d5c59d-l5g2b\" (UID: \"b04ec1dc-5e71-4fed-abc5-6875befad478\") " pod="openshift-ingress/router-default-944d5c59d-l5g2b"
Sep 16 12:40:36.405677 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.405468    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174-config\") pod \"console-operator-696884fb9c-vbt8d\" (UID: \"0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174\") " pod="openshift-console-operator/console-operator-696884fb9c-vbt8d"
Sep 16 12:40:36.405677 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.405503    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174-serving-cert\") pod \"console-operator-696884fb9c-vbt8d\" (UID: \"0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174\") " pod="openshift-console-operator/console-operator-696884fb9c-vbt8d"
Sep 16 12:40:36.405677 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.405633    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ad668dd0-69c8-435f-9b75-32f037bfdbc1-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5q4bl\" (UID: \"ad668dd0-69c8-435f-9b75-32f037bfdbc1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5q4bl"
Sep 16 12:40:36.405834 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.405674    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174-trusted-ca\") pod \"console-operator-696884fb9c-vbt8d\" (UID: \"0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174\") " pod="openshift-console-operator/console-operator-696884fb9c-vbt8d"
Sep 16 12:40:36.406171 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.406151    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174-config\") pod \"console-operator-696884fb9c-vbt8d\" (UID: \"0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174\") " pod="openshift-console-operator/console-operator-696884fb9c-vbt8d"
Sep 16 12:40:36.406477 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.406452    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ad668dd0-69c8-435f-9b75-32f037bfdbc1-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5q4bl\" (UID: \"ad668dd0-69c8-435f-9b75-32f037bfdbc1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5q4bl"
Sep 16 12:40:36.406551 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.406478    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174-trusted-ca\") pod \"console-operator-696884fb9c-vbt8d\" (UID: \"0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174\") " pod="openshift-console-operator/console-operator-696884fb9c-vbt8d"
Sep 16 12:40:36.407650 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.407629    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ad668dd0-69c8-435f-9b75-32f037bfdbc1-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5q4bl\" (UID: \"ad668dd0-69c8-435f-9b75-32f037bfdbc1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5q4bl"
Sep 16 12:40:36.407821 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.407804    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174-serving-cert\") pod \"console-operator-696884fb9c-vbt8d\" (UID: \"0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174\") " pod="openshift-console-operator/console-operator-696884fb9c-vbt8d"
Sep 16 12:40:36.414243 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.414216    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6d6b\" (UniqueName: \"kubernetes.io/projected/ad668dd0-69c8-435f-9b75-32f037bfdbc1-kube-api-access-q6d6b\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5q4bl\" (UID: \"ad668dd0-69c8-435f-9b75-32f037bfdbc1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5q4bl"
Sep 16 12:40:36.414337 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.414323    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljbg4\" (UniqueName: \"kubernetes.io/projected/e0992b2d-fbf2-4e89-afbc-543ce47e0615-kube-api-access-ljbg4\") pod \"volume-data-source-validator-86695b7cc7-mhv8x\" (UID: \"e0992b2d-fbf2-4e89-afbc-543ce47e0615\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-mhv8x"
Sep 16 12:40:36.414628 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.414609    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq699\" (UniqueName: \"kubernetes.io/projected/9306d26d-8a34-4f8d-91da-68ba20f0afa0-kube-api-access-mq699\") pod \"network-check-source-6f8b768455-qfll2\" (UID: \"9306d26d-8a34-4f8d-91da-68ba20f0afa0\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-qfll2"
Sep 16 12:40:36.414878 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.414859    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x2q9\" (UniqueName: \"kubernetes.io/projected/0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174-kube-api-access-8x2q9\") pod \"console-operator-696884fb9c-vbt8d\" (UID: \"0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174\") " pod="openshift-console-operator/console-operator-696884fb9c-vbt8d"
Sep 16 12:40:36.484528 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.484501    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-qfll2"
Sep 16 12:40:36.492001 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.491984    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5q4bl"
Sep 16 12:40:36.499750 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.499731    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-mhv8x"
Sep 16 12:40:36.505614 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.505593    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-vbt8d"
Sep 16 12:40:36.506378 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.505971    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b04ec1dc-5e71-4fed-abc5-6875befad478-default-certificate\") pod \"router-default-944d5c59d-l5g2b\" (UID: \"b04ec1dc-5e71-4fed-abc5-6875befad478\") " pod="openshift-ingress/router-default-944d5c59d-l5g2b"
Sep 16 12:40:36.506378 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.506007    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b04ec1dc-5e71-4fed-abc5-6875befad478-stats-auth\") pod \"router-default-944d5c59d-l5g2b\" (UID: \"b04ec1dc-5e71-4fed-abc5-6875befad478\") " pod="openshift-ingress/router-default-944d5c59d-l5g2b"
Sep 16 12:40:36.506378 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.506086    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd4bs\" (UniqueName: \"kubernetes.io/projected/b04ec1dc-5e71-4fed-abc5-6875befad478-kube-api-access-hd4bs\") pod \"router-default-944d5c59d-l5g2b\" (UID: \"b04ec1dc-5e71-4fed-abc5-6875befad478\") " pod="openshift-ingress/router-default-944d5c59d-l5g2b"
Sep 16 12:40:36.506378 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.506118    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b04ec1dc-5e71-4fed-abc5-6875befad478-metrics-certs\") pod \"router-default-944d5c59d-l5g2b\" (UID: \"b04ec1dc-5e71-4fed-abc5-6875befad478\") " pod="openshift-ingress/router-default-944d5c59d-l5g2b"
Sep 16 12:40:36.506378 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.506142    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b04ec1dc-5e71-4fed-abc5-6875befad478-service-ca-bundle\") pod \"router-default-944d5c59d-l5g2b\" (UID: \"b04ec1dc-5e71-4fed-abc5-6875befad478\") " pod="openshift-ingress/router-default-944d5c59d-l5g2b"
Sep 16 12:40:36.506378 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:36.506312    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b04ec1dc-5e71-4fed-abc5-6875befad478-service-ca-bundle podName:b04ec1dc-5e71-4fed-abc5-6875befad478 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:37.006294215 +0000 UTC m=+143.453526862 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b04ec1dc-5e71-4fed-abc5-6875befad478-service-ca-bundle") pod "router-default-944d5c59d-l5g2b" (UID: "b04ec1dc-5e71-4fed-abc5-6875befad478") : configmap references non-existent config key: service-ca.crt
Sep 16 12:40:36.506935 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:36.506501    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 12:40:36.506935 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:36.506677    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b04ec1dc-5e71-4fed-abc5-6875befad478-metrics-certs podName:b04ec1dc-5e71-4fed-abc5-6875befad478 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:37.006620767 +0000 UTC m=+143.453853408 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b04ec1dc-5e71-4fed-abc5-6875befad478-metrics-certs") pod "router-default-944d5c59d-l5g2b" (UID: "b04ec1dc-5e71-4fed-abc5-6875befad478") : secret "router-metrics-certs-default" not found
Sep 16 12:40:36.509303 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.509251    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b04ec1dc-5e71-4fed-abc5-6875befad478-default-certificate\") pod \"router-default-944d5c59d-l5g2b\" (UID: \"b04ec1dc-5e71-4fed-abc5-6875befad478\") " pod="openshift-ingress/router-default-944d5c59d-l5g2b"
Sep 16 12:40:36.509695 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.509673    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b04ec1dc-5e71-4fed-abc5-6875befad478-stats-auth\") pod \"router-default-944d5c59d-l5g2b\" (UID: \"b04ec1dc-5e71-4fed-abc5-6875befad478\") " pod="openshift-ingress/router-default-944d5c59d-l5g2b"
Sep 16 12:40:36.515839 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.515784    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd4bs\" (UniqueName: \"kubernetes.io/projected/b04ec1dc-5e71-4fed-abc5-6875befad478-kube-api-access-hd4bs\") pod \"router-default-944d5c59d-l5g2b\" (UID: \"b04ec1dc-5e71-4fed-abc5-6875befad478\") " pod="openshift-ingress/router-default-944d5c59d-l5g2b"
Sep 16 12:40:36.763062 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.762994    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-qfll2"]
Sep 16 12:40:36.765781 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:40:36.765756    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9306d26d_8a34_4f8d_91da_68ba20f0afa0.slice/crio-ff57062250af714424a908b4fdba42a87eb17573e2c3a4cee7e7b3d0782fb3fe WatchSource:0}: Error finding container ff57062250af714424a908b4fdba42a87eb17573e2c3a4cee7e7b3d0782fb3fe: Status 404 returned error can't find the container with id ff57062250af714424a908b4fdba42a87eb17573e2c3a4cee7e7b3d0782fb3fe
Sep 16 12:40:36.793645 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.793618    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5q4bl"]
Sep 16 12:40:36.794757 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.794695    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-mhv8x"]
Sep 16 12:40:36.795863 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.795844    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-vbt8d"]
Sep 16 12:40:36.797119 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:40:36.797090    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad668dd0_69c8_435f_9b75_32f037bfdbc1.slice/crio-3e8d44916895da63d8ec63a635d4bfe3f36da1702232888b58d85cdb8d8ea027 WatchSource:0}: Error finding container 3e8d44916895da63d8ec63a635d4bfe3f36da1702232888b58d85cdb8d8ea027: Status 404 returned error can't find the container with id 3e8d44916895da63d8ec63a635d4bfe3f36da1702232888b58d85cdb8d8ea027
Sep 16 12:40:36.797439 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:40:36.797420    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0992b2d_fbf2_4e89_afbc_543ce47e0615.slice/crio-4856c64b2bfe44dd4f74739029693d08b93bf1420b59c7496ad65d0abce07419 WatchSource:0}: Error finding container 4856c64b2bfe44dd4f74739029693d08b93bf1420b59c7496ad65d0abce07419: Status 404 returned error can't find the container with id 4856c64b2bfe44dd4f74739029693d08b93bf1420b59c7496ad65d0abce07419
Sep 16 12:40:36.798640 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:40:36.798619    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab2d86c_9ac2_4b80_87c5_d5f6d98fb174.slice/crio-b02a3b3eb6d4daba7b1018961c0f30661963180836e3f207db6ba23a7c041bd1 WatchSource:0}: Error finding container b02a3b3eb6d4daba7b1018961c0f30661963180836e3f207db6ba23a7c041bd1: Status 404 returned error can't find the container with id b02a3b3eb6d4daba7b1018961c0f30661963180836e3f207db6ba23a7c041bd1
Sep 16 12:40:36.808587 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:36.808566    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a583c3d2-5168-4901-89ec-2c64993e60ec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-mlzkb\" (UID: \"a583c3d2-5168-4901-89ec-2c64993e60ec\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mlzkb"
Sep 16 12:40:36.808681 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:36.808671    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 12:40:36.808726 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:36.808714    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a583c3d2-5168-4901-89ec-2c64993e60ec-cluster-monitoring-operator-tls podName:a583c3d2-5168-4901-89ec-2c64993e60ec nodeName:}" failed. No retries permitted until 2026-09-16 12:40:37.80870003 +0000 UTC m=+144.255932650 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a583c3d2-5168-4901-89ec-2c64993e60ec-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-mlzkb" (UID: "a583c3d2-5168-4901-89ec-2c64993e60ec") : secret "cluster-monitoring-operator-tls" not found
Sep 16 12:40:37.010365 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:37.010329    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b04ec1dc-5e71-4fed-abc5-6875befad478-metrics-certs\") pod \"router-default-944d5c59d-l5g2b\" (UID: \"b04ec1dc-5e71-4fed-abc5-6875befad478\") " pod="openshift-ingress/router-default-944d5c59d-l5g2b"
Sep 16 12:40:37.010365 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:37.010367    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b04ec1dc-5e71-4fed-abc5-6875befad478-service-ca-bundle\") pod \"router-default-944d5c59d-l5g2b\" (UID: \"b04ec1dc-5e71-4fed-abc5-6875befad478\") " pod="openshift-ingress/router-default-944d5c59d-l5g2b"
Sep 16 12:40:37.010607 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:37.010477    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b04ec1dc-5e71-4fed-abc5-6875befad478-service-ca-bundle podName:b04ec1dc-5e71-4fed-abc5-6875befad478 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:38.010462031 +0000 UTC m=+144.457694651 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b04ec1dc-5e71-4fed-abc5-6875befad478-service-ca-bundle") pod "router-default-944d5c59d-l5g2b" (UID: "b04ec1dc-5e71-4fed-abc5-6875befad478") : configmap references non-existent config key: service-ca.crt
Sep 16 12:40:37.010607 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:37.010505    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 12:40:37.010607 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:37.010572    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b04ec1dc-5e71-4fed-abc5-6875befad478-metrics-certs podName:b04ec1dc-5e71-4fed-abc5-6875befad478 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:38.010554888 +0000 UTC m=+144.457787529 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b04ec1dc-5e71-4fed-abc5-6875befad478-metrics-certs") pod "router-default-944d5c59d-l5g2b" (UID: "b04ec1dc-5e71-4fed-abc5-6875befad478") : secret "router-metrics-certs-default" not found
Sep 16 12:40:37.500941 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:37.500900    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-mhv8x" event={"ID":"e0992b2d-fbf2-4e89-afbc-543ce47e0615","Type":"ContainerStarted","Data":"4856c64b2bfe44dd4f74739029693d08b93bf1420b59c7496ad65d0abce07419"}
Sep 16 12:40:37.502347 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:37.502297    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5q4bl" event={"ID":"ad668dd0-69c8-435f-9b75-32f037bfdbc1","Type":"ContainerStarted","Data":"3e8d44916895da63d8ec63a635d4bfe3f36da1702232888b58d85cdb8d8ea027"}
Sep 16 12:40:37.505183 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:37.505157    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-qfll2" event={"ID":"9306d26d-8a34-4f8d-91da-68ba20f0afa0","Type":"ContainerStarted","Data":"db20376c35d29f4371f4f23136fcf566efc96b71eb968b45a3e412911093665c"}
Sep 16 12:40:37.505268 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:37.505201    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-qfll2" event={"ID":"9306d26d-8a34-4f8d-91da-68ba20f0afa0","Type":"ContainerStarted","Data":"ff57062250af714424a908b4fdba42a87eb17573e2c3a4cee7e7b3d0782fb3fe"}
Sep 16 12:40:37.506906 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:37.506886    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-vbt8d" event={"ID":"0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174","Type":"ContainerStarted","Data":"b02a3b3eb6d4daba7b1018961c0f30661963180836e3f207db6ba23a7c041bd1"}
Sep 16 12:40:37.525477 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:37.524102    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-qfll2" podStartSLOduration=1.5240826090000001 podStartE2EDuration="1.524082609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:40:37.523064755 +0000 UTC m=+143.970297397" watchObservedRunningTime="2026-09-16 12:40:37.524082609 +0000 UTC m=+143.971315250"
Sep 16 12:40:37.818641 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:37.818608    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a583c3d2-5168-4901-89ec-2c64993e60ec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-mlzkb\" (UID: \"a583c3d2-5168-4901-89ec-2c64993e60ec\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mlzkb"
Sep 16 12:40:37.818782 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:37.818763    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 12:40:37.818845 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:37.818826    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a583c3d2-5168-4901-89ec-2c64993e60ec-cluster-monitoring-operator-tls podName:a583c3d2-5168-4901-89ec-2c64993e60ec nodeName:}" failed. No retries permitted until 2026-09-16 12:40:39.818811585 +0000 UTC m=+146.266044205 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a583c3d2-5168-4901-89ec-2c64993e60ec-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-mlzkb" (UID: "a583c3d2-5168-4901-89ec-2c64993e60ec") : secret "cluster-monitoring-operator-tls" not found
Sep 16 12:40:38.021688 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:38.020954    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b04ec1dc-5e71-4fed-abc5-6875befad478-metrics-certs\") pod \"router-default-944d5c59d-l5g2b\" (UID: \"b04ec1dc-5e71-4fed-abc5-6875befad478\") " pod="openshift-ingress/router-default-944d5c59d-l5g2b"
Sep 16 12:40:38.021688 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:38.021003    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b04ec1dc-5e71-4fed-abc5-6875befad478-service-ca-bundle\") pod \"router-default-944d5c59d-l5g2b\" (UID: \"b04ec1dc-5e71-4fed-abc5-6875befad478\") " pod="openshift-ingress/router-default-944d5c59d-l5g2b"
Sep 16 12:40:38.021688 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:38.021190    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b04ec1dc-5e71-4fed-abc5-6875befad478-service-ca-bundle podName:b04ec1dc-5e71-4fed-abc5-6875befad478 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:40.02116886 +0000 UTC m=+146.468401502 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b04ec1dc-5e71-4fed-abc5-6875befad478-service-ca-bundle") pod "router-default-944d5c59d-l5g2b" (UID: "b04ec1dc-5e71-4fed-abc5-6875befad478") : configmap references non-existent config key: service-ca.crt
Sep 16 12:40:38.021688 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:38.021277    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 12:40:38.021688 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:38.021312    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b04ec1dc-5e71-4fed-abc5-6875befad478-metrics-certs podName:b04ec1dc-5e71-4fed-abc5-6875befad478 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:40.021301339 +0000 UTC m=+146.468533975 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b04ec1dc-5e71-4fed-abc5-6875befad478-metrics-certs") pod "router-default-944d5c59d-l5g2b" (UID: "b04ec1dc-5e71-4fed-abc5-6875befad478") : secret "router-metrics-certs-default" not found
Sep 16 12:40:38.514354 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:38.514314    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-mhv8x" event={"ID":"e0992b2d-fbf2-4e89-afbc-543ce47e0615","Type":"ContainerStarted","Data":"ec192b784a0cd588ff4af7bfd19010db74a03ae268dfe87bc3446ab3a2affd74"}
Sep 16 12:40:38.532087 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:38.532021    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-mhv8x" podStartSLOduration=1.57345453 podStartE2EDuration="2.532007188s" totalImagesPullingTime="958.552658ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:40:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:40:38.530979273 +0000 UTC m=+144.978211926" watchObservedRunningTime="2026-09-16 12:40:38.532007188 +0000 UTC m=+144.979239830"
Sep 16 12:40:39.519219 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:39.519192    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vbt8d_0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174/console-operator/0.log"
Sep 16 12:40:39.519623 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:39.519235    2577 generic.go:356] "Generic (PLEG): container finished" podID="0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174" containerID="953e222ea367a46a9653a08b53bda879587867a0c09646cfe11da06cd739c70f" exitCode=255
Sep 16 12:40:39.519623 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:39.519326    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-vbt8d" event={"ID":"0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174","Type":"ContainerDied","Data":"953e222ea367a46a9653a08b53bda879587867a0c09646cfe11da06cd739c70f"}
Sep 16 12:40:39.519623 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:39.519545    2577 scope.go:117] "RemoveContainer" containerID="953e222ea367a46a9653a08b53bda879587867a0c09646cfe11da06cd739c70f"
Sep 16 12:40:39.520891 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:39.520868    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5q4bl" event={"ID":"ad668dd0-69c8-435f-9b75-32f037bfdbc1","Type":"ContainerStarted","Data":"6d4bf62262946a247654cbbc0417c5eb5db38c22267ca6750e64e19acdd340b5"}
Sep 16 12:40:39.555158 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:39.555118    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5q4bl" podStartSLOduration=1.212526346 podStartE2EDuration="3.555102977s" totalImagesPullingTime="2.342576631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:40:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:40:39.553775343 +0000 UTC m=+146.001007988" watchObservedRunningTime="2026-09-16 12:40:39.555102977 +0000 UTC m=+146.002335662"
Sep 16 12:40:39.836773 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:39.836690    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a583c3d2-5168-4901-89ec-2c64993e60ec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-mlzkb\" (UID: \"a583c3d2-5168-4901-89ec-2c64993e60ec\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mlzkb"
Sep 16 12:40:39.836912 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:39.836799    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 12:40:39.836912 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:39.836853    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a583c3d2-5168-4901-89ec-2c64993e60ec-cluster-monitoring-operator-tls podName:a583c3d2-5168-4901-89ec-2c64993e60ec nodeName:}" failed. No retries permitted until 2026-09-16 12:40:43.836839966 +0000 UTC m=+150.284072585 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a583c3d2-5168-4901-89ec-2c64993e60ec-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-mlzkb" (UID: "a583c3d2-5168-4901-89ec-2c64993e60ec") : secret "cluster-monitoring-operator-tls" not found
Sep 16 12:40:40.038319 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:40.038282    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b04ec1dc-5e71-4fed-abc5-6875befad478-metrics-certs\") pod \"router-default-944d5c59d-l5g2b\" (UID: \"b04ec1dc-5e71-4fed-abc5-6875befad478\") " pod="openshift-ingress/router-default-944d5c59d-l5g2b"
Sep 16 12:40:40.038319 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:40.038321    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b04ec1dc-5e71-4fed-abc5-6875befad478-service-ca-bundle\") pod \"router-default-944d5c59d-l5g2b\" (UID: \"b04ec1dc-5e71-4fed-abc5-6875befad478\") " pod="openshift-ingress/router-default-944d5c59d-l5g2b"
Sep 16 12:40:40.038553 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:40.038462    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 12:40:40.038553 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:40.038471    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b04ec1dc-5e71-4fed-abc5-6875befad478-service-ca-bundle podName:b04ec1dc-5e71-4fed-abc5-6875befad478 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:44.038456789 +0000 UTC m=+150.485689409 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b04ec1dc-5e71-4fed-abc5-6875befad478-service-ca-bundle") pod "router-default-944d5c59d-l5g2b" (UID: "b04ec1dc-5e71-4fed-abc5-6875befad478") : configmap references non-existent config key: service-ca.crt
Sep 16 12:40:40.038553 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:40.038537    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b04ec1dc-5e71-4fed-abc5-6875befad478-metrics-certs podName:b04ec1dc-5e71-4fed-abc5-6875befad478 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:44.038518418 +0000 UTC m=+150.485751049 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b04ec1dc-5e71-4fed-abc5-6875befad478-metrics-certs") pod "router-default-944d5c59d-l5g2b" (UID: "b04ec1dc-5e71-4fed-abc5-6875befad478") : secret "router-metrics-certs-default" not found
Sep 16 12:40:40.525519 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:40.525490    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vbt8d_0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174/console-operator/1.log"
Sep 16 12:40:40.525932 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:40.525915    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vbt8d_0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174/console-operator/0.log"
Sep 16 12:40:40.526012 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:40.525951    2577 generic.go:356] "Generic (PLEG): container finished" podID="0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174" containerID="095571d3bead20a301fe9d99a155fc94eb2ddee2ac7b7602eac8ec41ba9c8428" exitCode=255
Sep 16 12:40:40.526095 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:40.526071    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-vbt8d" event={"ID":"0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174","Type":"ContainerDied","Data":"095571d3bead20a301fe9d99a155fc94eb2ddee2ac7b7602eac8ec41ba9c8428"}
Sep 16 12:40:40.526148 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:40.526136    2577 scope.go:117] "RemoveContainer" containerID="953e222ea367a46a9653a08b53bda879587867a0c09646cfe11da06cd739c70f"
Sep 16 12:40:40.526394 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:40.526369    2577 scope.go:117] "RemoveContainer" containerID="095571d3bead20a301fe9d99a155fc94eb2ddee2ac7b7602eac8ec41ba9c8428"
Sep 16 12:40:40.526587 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:40.526565    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-vbt8d_openshift-console-operator(0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174)\"" pod="openshift-console-operator/console-operator-696884fb9c-vbt8d" podUID="0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174"
Sep 16 12:40:41.119709 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:41.119679    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nps4n"]
Sep 16 12:40:41.123900 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:41.123883    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nps4n"
Sep 16 12:40:41.126528 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:41.126506    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 12:40:41.126909 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:41.126893    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 12:40:41.127471 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:41.127443    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 12:40:41.127765 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:41.127750    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 12:40:41.127907 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:41.127894    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9z4xc\""
Sep 16 12:40:41.137751 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:41.137723    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nps4n"]
Sep 16 12:40:41.248926 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:41.248892    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-data-volume\") pod \"insights-runtime-extractor-nps4n\" (UID: \"fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7\") " pod="openshift-insights/insights-runtime-extractor-nps4n"
Sep 16 12:40:41.249088 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:41.248940    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-crio-socket\") pod \"insights-runtime-extractor-nps4n\" (UID: \"fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7\") " pod="openshift-insights/insights-runtime-extractor-nps4n"
Sep 16 12:40:41.249088 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:41.248978    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nps4n\" (UID: \"fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7\") " pod="openshift-insights/insights-runtime-extractor-nps4n"
Sep 16 12:40:41.249088 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:41.249004    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nps4n\" (UID: \"fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7\") " pod="openshift-insights/insights-runtime-extractor-nps4n"
Sep 16 12:40:41.249088 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:41.249045    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrlx9\" (UniqueName: \"kubernetes.io/projected/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-kube-api-access-hrlx9\") pod \"insights-runtime-extractor-nps4n\" (UID: \"fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7\") " pod="openshift-insights/insights-runtime-extractor-nps4n"
Sep 16 12:40:41.349895 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:41.349863    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-data-volume\") pod \"insights-runtime-extractor-nps4n\" (UID: \"fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7\") " pod="openshift-insights/insights-runtime-extractor-nps4n"
Sep 16 12:40:41.349895 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:41.349900    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-crio-socket\") pod \"insights-runtime-extractor-nps4n\" (UID: \"fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7\") " pod="openshift-insights/insights-runtime-extractor-nps4n"
Sep 16 12:40:41.350196 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:41.349918    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nps4n\" (UID: \"fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7\") " pod="openshift-insights/insights-runtime-extractor-nps4n"
Sep 16 12:40:41.350196 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:41.349935    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nps4n\" (UID: \"fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7\") " pod="openshift-insights/insights-runtime-extractor-nps4n"
Sep 16 12:40:41.350196 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:41.350001    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-crio-socket\") pod \"insights-runtime-extractor-nps4n\" (UID: \"fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7\") " pod="openshift-insights/insights-runtime-extractor-nps4n"
Sep 16 12:40:41.350196 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:41.350017    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:40:41.350196 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:41.350020    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:40:41.350196 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:41.350069    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrlx9\" (UniqueName: \"kubernetes.io/projected/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-kube-api-access-hrlx9\") pod \"insights-runtime-extractor-nps4n\" (UID: \"fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7\") " pod="openshift-insights/insights-runtime-extractor-nps4n"
Sep 16 12:40:41.350196 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:41.350104    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-kube-rbac-proxy-cm podName:fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:41.850083076 +0000 UTC m=+148.297315710 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nps4n" (UID: "fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7") : configmap "kube-rbac-proxy" not found
Sep 16 12:40:41.350196 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:41.350183    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-insights-runtime-extractor-tls podName:fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:41.850155658 +0000 UTC m=+148.297388278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nps4n" (UID: "fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:40:41.350498 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:41.350261    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-data-volume\") pod \"insights-runtime-extractor-nps4n\" (UID: \"fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7\") " pod="openshift-insights/insights-runtime-extractor-nps4n"
Sep 16 12:40:41.358890 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:41.358872    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrlx9\" (UniqueName: \"kubernetes.io/projected/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-kube-api-access-hrlx9\") pod \"insights-runtime-extractor-nps4n\" (UID: \"fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7\") " pod="openshift-insights/insights-runtime-extractor-nps4n"
Sep 16 12:40:41.530225 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:41.530201    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vbt8d_0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174/console-operator/1.log"
Sep 16 12:40:41.530567 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:41.530533    2577 scope.go:117] "RemoveContainer" containerID="095571d3bead20a301fe9d99a155fc94eb2ddee2ac7b7602eac8ec41ba9c8428"
Sep 16 12:40:41.530694 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:41.530676    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-vbt8d_openshift-console-operator(0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174)\"" pod="openshift-console-operator/console-operator-696884fb9c-vbt8d" podUID="0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174"
Sep 16 12:40:41.854633 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:41.854528    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nps4n\" (UID: \"fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7\") " pod="openshift-insights/insights-runtime-extractor-nps4n"
Sep 16 12:40:41.854633 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:41.854581    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nps4n\" (UID: \"fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7\") " pod="openshift-insights/insights-runtime-extractor-nps4n"
Sep 16 12:40:41.854832 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:41.854650    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:40:41.854832 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:41.854717    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-kube-rbac-proxy-cm podName:fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:42.85470089 +0000 UTC m=+149.301933509 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nps4n" (UID: "fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7") : configmap "kube-rbac-proxy" not found
Sep 16 12:40:41.854832 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:41.854730    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:40:41.854832 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:41.854786    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-insights-runtime-extractor-tls podName:fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:42.854775902 +0000 UTC m=+149.302008522 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nps4n" (UID: "fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:40:42.862328 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:42.862294    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nps4n\" (UID: \"fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7\") " pod="openshift-insights/insights-runtime-extractor-nps4n"
Sep 16 12:40:42.862736 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:42.862332    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nps4n\" (UID: \"fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7\") " pod="openshift-insights/insights-runtime-extractor-nps4n"
Sep 16 12:40:42.862736 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:42.862417    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:40:42.862736 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:42.862481    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-kube-rbac-proxy-cm podName:fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:44.86246457 +0000 UTC m=+151.309697215 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nps4n" (UID: "fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7") : configmap "kube-rbac-proxy" not found
Sep 16 12:40:42.862736 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:42.862502    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:40:42.862736 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:42.862554    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-insights-runtime-extractor-tls podName:fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:44.862539133 +0000 UTC m=+151.309771781 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nps4n" (UID: "fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:40:43.870712 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:43.870676    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a583c3d2-5168-4901-89ec-2c64993e60ec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-mlzkb\" (UID: \"a583c3d2-5168-4901-89ec-2c64993e60ec\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mlzkb"
Sep 16 12:40:43.871130 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:43.870789    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 12:40:43.871130 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:43.870840    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a583c3d2-5168-4901-89ec-2c64993e60ec-cluster-monitoring-operator-tls podName:a583c3d2-5168-4901-89ec-2c64993e60ec nodeName:}" failed. No retries permitted until 2026-09-16 12:40:51.870826492 +0000 UTC m=+158.318059113 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a583c3d2-5168-4901-89ec-2c64993e60ec-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-mlzkb" (UID: "a583c3d2-5168-4901-89ec-2c64993e60ec") : secret "cluster-monitoring-operator-tls" not found
Sep 16 12:40:43.987432 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:43.987404    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t47mb_c44032b0-4e54-42d9-945f-7d401e20d50c/dns-node-resolver/0.log"
Sep 16 12:40:44.072240 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:44.072209    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b04ec1dc-5e71-4fed-abc5-6875befad478-metrics-certs\") pod \"router-default-944d5c59d-l5g2b\" (UID: \"b04ec1dc-5e71-4fed-abc5-6875befad478\") " pod="openshift-ingress/router-default-944d5c59d-l5g2b"
Sep 16 12:40:44.072240 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:44.072240    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b04ec1dc-5e71-4fed-abc5-6875befad478-service-ca-bundle\") pod \"router-default-944d5c59d-l5g2b\" (UID: \"b04ec1dc-5e71-4fed-abc5-6875befad478\") " pod="openshift-ingress/router-default-944d5c59d-l5g2b"
Sep 16 12:40:44.072424 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:44.072342    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 12:40:44.072424 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:44.072365    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b04ec1dc-5e71-4fed-abc5-6875befad478-service-ca-bundle podName:b04ec1dc-5e71-4fed-abc5-6875befad478 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:52.072350732 +0000 UTC m=+158.519583366 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b04ec1dc-5e71-4fed-abc5-6875befad478-service-ca-bundle") pod "router-default-944d5c59d-l5g2b" (UID: "b04ec1dc-5e71-4fed-abc5-6875befad478") : configmap references non-existent config key: service-ca.crt
Sep 16 12:40:44.072424 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:44.072397    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b04ec1dc-5e71-4fed-abc5-6875befad478-metrics-certs podName:b04ec1dc-5e71-4fed-abc5-6875befad478 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:52.072384255 +0000 UTC m=+158.519616875 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b04ec1dc-5e71-4fed-abc5-6875befad478-metrics-certs") pod "router-default-944d5c59d-l5g2b" (UID: "b04ec1dc-5e71-4fed-abc5-6875befad478") : secret "router-metrics-certs-default" not found
Sep 16 12:40:44.787084 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:44.787057    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8bxln_d3f21250-f182-428a-a11e-99f2a158ca61/node-ca/0.log"
Sep 16 12:40:44.878751 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:44.878717    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nps4n\" (UID: \"fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7\") " pod="openshift-insights/insights-runtime-extractor-nps4n"
Sep 16 12:40:44.879164 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:44.878757    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nps4n\" (UID: \"fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7\") " pod="openshift-insights/insights-runtime-extractor-nps4n"
Sep 16 12:40:44.879164 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:44.878832    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:40:44.879164 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:44.878890    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-kube-rbac-proxy-cm podName:fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:48.878875101 +0000 UTC m=+155.326107720 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nps4n" (UID: "fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7") : configmap "kube-rbac-proxy" not found
Sep 16 12:40:44.879164 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:44.878903    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:40:44.879164 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:44.878951    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-insights-runtime-extractor-tls podName:fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:48.878936806 +0000 UTC m=+155.326169443 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nps4n" (UID: "fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:40:45.986731 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:45.986702    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pjjgz_6bd5627d-472c-43bf-a550-8f30201dee34/serve-healthcheck-canary/0.log"
Sep 16 12:40:46.506178 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:46.506129    2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-vbt8d"
Sep 16 12:40:46.506178 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:46.506194    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-vbt8d"
Sep 16 12:40:46.506537 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:46.506525    2577 scope.go:117] "RemoveContainer" containerID="095571d3bead20a301fe9d99a155fc94eb2ddee2ac7b7602eac8ec41ba9c8428"
Sep 16 12:40:46.506680 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:46.506663    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-vbt8d_openshift-console-operator(0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174)\"" pod="openshift-console-operator/console-operator-696884fb9c-vbt8d" podUID="0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174"
Sep 16 12:40:46.587521 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:46.587488    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-5q4bl_ad668dd0-69c8-435f-9b75-32f037bfdbc1/kube-storage-version-migrator-operator/0.log"
Sep 16 12:40:48.913858 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:48.913823    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nps4n\" (UID: \"fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7\") " pod="openshift-insights/insights-runtime-extractor-nps4n"
Sep 16 12:40:48.913858 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:48.913861    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nps4n\" (UID: \"fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7\") " pod="openshift-insights/insights-runtime-extractor-nps4n"
Sep 16 12:40:48.914317 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:48.913939    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:40:48.914317 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:48.913973    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:40:48.914317 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:48.914001    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-kube-rbac-proxy-cm podName:fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:56.913984339 +0000 UTC m=+163.361216959 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nps4n" (UID: "fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7") : configmap "kube-rbac-proxy" not found
Sep 16 12:40:48.914317 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:48.914048    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-insights-runtime-extractor-tls podName:fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:56.914011179 +0000 UTC m=+163.361243823 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nps4n" (UID: "fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:40:51.937697 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:51.937657    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a583c3d2-5168-4901-89ec-2c64993e60ec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-mlzkb\" (UID: \"a583c3d2-5168-4901-89ec-2c64993e60ec\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mlzkb"
Sep 16 12:40:51.938086 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:51.937799    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 12:40:51.938086 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:51.937863    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a583c3d2-5168-4901-89ec-2c64993e60ec-cluster-monitoring-operator-tls podName:a583c3d2-5168-4901-89ec-2c64993e60ec nodeName:}" failed. No retries permitted until 2026-09-16 12:41:07.937845593 +0000 UTC m=+174.385078218 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a583c3d2-5168-4901-89ec-2c64993e60ec-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-mlzkb" (UID: "a583c3d2-5168-4901-89ec-2c64993e60ec") : secret "cluster-monitoring-operator-tls" not found
Sep 16 12:40:51.972466 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:51.972428    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m" podUID="70c5a239-65d0-4d57-861b-0ba00290c8bf"
Sep 16 12:40:51.989724 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:51.989690    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-ct9vp" podUID="4f5ad86c-a72b-4f7e-aa27-484b2e8a3261"
Sep 16 12:40:52.064342 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:52.064309    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-tmfpk" podUID="aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5"
Sep 16 12:40:52.138466 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:52.138443    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b04ec1dc-5e71-4fed-abc5-6875befad478-metrics-certs\") pod \"router-default-944d5c59d-l5g2b\" (UID: \"b04ec1dc-5e71-4fed-abc5-6875befad478\") " pod="openshift-ingress/router-default-944d5c59d-l5g2b"
Sep 16 12:40:52.138609 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:52.138473    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b04ec1dc-5e71-4fed-abc5-6875befad478-service-ca-bundle\") pod \"router-default-944d5c59d-l5g2b\" (UID: \"b04ec1dc-5e71-4fed-abc5-6875befad478\") " pod="openshift-ingress/router-default-944d5c59d-l5g2b"
Sep 16 12:40:52.138609 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:52.138590    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 12:40:52.138693 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:52.138599    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b04ec1dc-5e71-4fed-abc5-6875befad478-service-ca-bundle podName:b04ec1dc-5e71-4fed-abc5-6875befad478 nodeName:}" failed. No retries permitted until 2026-09-16 12:41:08.138586169 +0000 UTC m=+174.585818789 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b04ec1dc-5e71-4fed-abc5-6875befad478-service-ca-bundle") pod "router-default-944d5c59d-l5g2b" (UID: "b04ec1dc-5e71-4fed-abc5-6875befad478") : configmap references non-existent config key: service-ca.crt
Sep 16 12:40:52.138693 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:52.138650    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b04ec1dc-5e71-4fed-abc5-6875befad478-metrics-certs podName:b04ec1dc-5e71-4fed-abc5-6875befad478 nodeName:}" failed. No retries permitted until 2026-09-16 12:41:08.138637164 +0000 UTC m=+174.585869784 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b04ec1dc-5e71-4fed-abc5-6875befad478-metrics-certs") pod "router-default-944d5c59d-l5g2b" (UID: "b04ec1dc-5e71-4fed-abc5-6875befad478") : secret "router-metrics-certs-default" not found
Sep 16 12:40:52.564307 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:52.564279    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ct9vp"
Sep 16 12:40:52.564502 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:52.564280    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m"
Sep 16 12:40:56.876899 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:56.876866    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f5ad86c-a72b-4f7e-aa27-484b2e8a3261-metrics-tls\") pod \"dns-default-ct9vp\" (UID: \"4f5ad86c-a72b-4f7e-aa27-484b2e8a3261\") " pod="openshift-dns/dns-default-ct9vp"
Sep 16 12:40:56.877334 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:56.876917    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-registry-tls\") pod \"image-registry-6fd89fff9d-lkz8m\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") " pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m"
Sep 16 12:40:56.879423 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:56.879394    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f5ad86c-a72b-4f7e-aa27-484b2e8a3261-metrics-tls\") pod \"dns-default-ct9vp\" (UID: \"4f5ad86c-a72b-4f7e-aa27-484b2e8a3261\") " pod="openshift-dns/dns-default-ct9vp"
Sep 16 12:40:56.879535 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:56.879484    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-registry-tls\") pod \"image-registry-6fd89fff9d-lkz8m\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") " pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m"
Sep 16 12:40:56.978038 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:56.978008    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nps4n\" (UID: \"fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7\") " pod="openshift-insights/insights-runtime-extractor-nps4n"
Sep 16 12:40:56.978226 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:56.978142    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nps4n\" (UID: \"fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7\") " pod="openshift-insights/insights-runtime-extractor-nps4n"
Sep 16 12:40:56.978226 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:56.978182    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:40:56.978339 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:40:56.978252    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-insights-runtime-extractor-tls podName:fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7 nodeName:}" failed. No retries permitted until 2026-09-16 12:41:12.978231803 +0000 UTC m=+179.425464440 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nps4n" (UID: "fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:40:56.978637 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:56.978618    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nps4n\" (UID: \"fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7\") " pod="openshift-insights/insights-runtime-extractor-nps4n"
Sep 16 12:40:57.069417 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:57.069386    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5ps7f\""
Sep 16 12:40:57.069557 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:57.069526    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fgm9v\""
Sep 16 12:40:57.075781 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:57.075760    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m"
Sep 16 12:40:57.075881 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:57.075789    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ct9vp"
Sep 16 12:40:57.358062 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:57.358017    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ct9vp"]
Sep 16 12:40:57.360661 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:57.360640    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6fd89fff9d-lkz8m"]
Sep 16 12:40:57.360981 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:40:57.360958    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f5ad86c_a72b_4f7e_aa27_484b2e8a3261.slice/crio-059e25c95c4dfb1dc0b4861a58287a90e1d993283037550156cd6ef5404c5c40 WatchSource:0}: Error finding container 059e25c95c4dfb1dc0b4861a58287a90e1d993283037550156cd6ef5404c5c40: Status 404 returned error can't find the container with id 059e25c95c4dfb1dc0b4861a58287a90e1d993283037550156cd6ef5404c5c40
Sep 16 12:40:57.364067 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:40:57.364042    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70c5a239_65d0_4d57_861b_0ba00290c8bf.slice/crio-401d5dede241162cf747cbf5c553bce3334bcb9709444c167bfe1e452d771067 WatchSource:0}: Error finding container 401d5dede241162cf747cbf5c553bce3334bcb9709444c167bfe1e452d771067: Status 404 returned error can't find the container with id 401d5dede241162cf747cbf5c553bce3334bcb9709444c167bfe1e452d771067
Sep 16 12:40:57.581349 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:57.581272    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m" event={"ID":"70c5a239-65d0-4d57-861b-0ba00290c8bf","Type":"ContainerStarted","Data":"7b483a1bc659962bb06f6091c8ccd8ccfeb0bef1ee390927aa2db7f6f258e3d3"}
Sep 16 12:40:57.581526 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:57.581362    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m" event={"ID":"70c5a239-65d0-4d57-861b-0ba00290c8bf","Type":"ContainerStarted","Data":"401d5dede241162cf747cbf5c553bce3334bcb9709444c167bfe1e452d771067"}
Sep 16 12:40:57.581526 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:57.581456    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m"
Sep 16 12:40:57.582494 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:57.582473    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ct9vp" event={"ID":"4f5ad86c-a72b-4f7e-aa27-484b2e8a3261","Type":"ContainerStarted","Data":"059e25c95c4dfb1dc0b4861a58287a90e1d993283037550156cd6ef5404c5c40"}
Sep 16 12:40:57.620047 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:57.619984    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m" podStartSLOduration=162.619968575 podStartE2EDuration="2m42.619968575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:38:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:40:57.619349478 +0000 UTC m=+164.066582121" watchObservedRunningTime="2026-09-16 12:40:57.619968575 +0000 UTC m=+164.067201217"
Sep 16 12:40:58.049056 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:58.049002    2577 scope.go:117] "RemoveContainer" containerID="095571d3bead20a301fe9d99a155fc94eb2ddee2ac7b7602eac8ec41ba9c8428"
Sep 16 12:40:58.587965 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:58.587934    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vbt8d_0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174/console-operator/1.log"
Sep 16 12:40:58.588138 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:58.588070    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-vbt8d" event={"ID":"0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174","Type":"ContainerStarted","Data":"9af6e734abda4fb62a68f0daa3373cfade5b78b48eaec525d8eb5534722fa3db"}
Sep 16 12:40:58.588630 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:58.588610    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-vbt8d"
Sep 16 12:40:58.608795 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:58.608754    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-vbt8d" podStartSLOduration=20.269794972 podStartE2EDuration="22.608738385s" totalImagesPullingTime="2.338943413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:40:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:40:58.60804127 +0000 UTC m=+165.055273902" watchObservedRunningTime="2026-09-16 12:40:58.608738385 +0000 UTC m=+165.055971027"
Sep 16 12:40:58.693697 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:58.693659    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-vbt8d"
Sep 16 12:40:59.592748 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:59.592664    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ct9vp" event={"ID":"4f5ad86c-a72b-4f7e-aa27-484b2e8a3261","Type":"ContainerStarted","Data":"8a047c6ce7b4297160f83e2219b6472e65a5ef68f24b20d8a922310887620171"}
Sep 16 12:40:59.592748 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:59.592716    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ct9vp" event={"ID":"4f5ad86c-a72b-4f7e-aa27-484b2e8a3261","Type":"ContainerStarted","Data":"acacc41363edfea783a51febc114e3e678051805f9fb41c9a49bf5f52953db16"}
Sep 16 12:40:59.592748 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:59.592749    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ct9vp"
Sep 16 12:40:59.613773 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:40:59.613722    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ct9vp" podStartSLOduration=129.719727358 podStartE2EDuration="2m11.613707658s" totalImagesPullingTime="1.8939803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:38:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:40:59.612062112 +0000 UTC m=+166.059294747" watchObservedRunningTime="2026-09-16 12:40:59.613707658 +0000 UTC m=+166.060940320"
Sep 16 12:41:04.674242 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:04.674214    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-kl8tl"]
Sep 16 12:41:04.677021 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:04.677004    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-kl8tl"
Sep 16 12:41:04.680112 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:04.680094    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 12:41:04.680749 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:04.680729    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 12:41:04.680881 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:04.680867    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-sgbvg\""
Sep 16 12:41:04.697839 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:04.697803    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-kl8tl"]
Sep 16 12:41:04.738966 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:04.738934    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcqbf\" (UniqueName: \"kubernetes.io/projected/c41aa457-55d7-4f45-a1b8-f42d4597182c-kube-api-access-hcqbf\") pod \"downloads-76f99584f4-kl8tl\" (UID: \"c41aa457-55d7-4f45-a1b8-f42d4597182c\") " pod="openshift-console/downloads-76f99584f4-kl8tl"
Sep 16 12:41:04.839704 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:04.839668    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcqbf\" (UniqueName: \"kubernetes.io/projected/c41aa457-55d7-4f45-a1b8-f42d4597182c-kube-api-access-hcqbf\") pod \"downloads-76f99584f4-kl8tl\" (UID: \"c41aa457-55d7-4f45-a1b8-f42d4597182c\") " pod="openshift-console/downloads-76f99584f4-kl8tl"
Sep 16 12:41:04.850929 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:04.850897    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcqbf\" (UniqueName: \"kubernetes.io/projected/c41aa457-55d7-4f45-a1b8-f42d4597182c-kube-api-access-hcqbf\") pod \"downloads-76f99584f4-kl8tl\" (UID: \"c41aa457-55d7-4f45-a1b8-f42d4597182c\") " pod="openshift-console/downloads-76f99584f4-kl8tl"
Sep 16 12:41:04.986187 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:04.986109    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-kl8tl"
Sep 16 12:41:05.258164 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:05.258132    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-kl8tl"]
Sep 16 12:41:05.261921 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:41:05.261893    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc41aa457_55d7_4f45_a1b8_f42d4597182c.slice/crio-fe028b6d88ae42d99bd5fdf743f5b73f15f6c9d2641daa93c641b6e430c60fa3 WatchSource:0}: Error finding container fe028b6d88ae42d99bd5fdf743f5b73f15f6c9d2641daa93c641b6e430c60fa3: Status 404 returned error can't find the container with id fe028b6d88ae42d99bd5fdf743f5b73f15f6c9d2641daa93c641b6e430c60fa3
Sep 16 12:41:05.611239 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:05.611132    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-kl8tl" event={"ID":"c41aa457-55d7-4f45-a1b8-f42d4597182c","Type":"ContainerStarted","Data":"fe028b6d88ae42d99bd5fdf743f5b73f15f6c9d2641daa93c641b6e430c60fa3"}
Sep 16 12:41:06.048821 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:06.048789    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmfpk"
Sep 16 12:41:07.970535 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:07.970501    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a583c3d2-5168-4901-89ec-2c64993e60ec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-mlzkb\" (UID: \"a583c3d2-5168-4901-89ec-2c64993e60ec\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mlzkb"
Sep 16 12:41:07.973240 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:07.973216    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a583c3d2-5168-4901-89ec-2c64993e60ec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-mlzkb\" (UID: \"a583c3d2-5168-4901-89ec-2c64993e60ec\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mlzkb"
Sep 16 12:41:08.171792 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:08.171760    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b04ec1dc-5e71-4fed-abc5-6875befad478-metrics-certs\") pod \"router-default-944d5c59d-l5g2b\" (UID: \"b04ec1dc-5e71-4fed-abc5-6875befad478\") " pod="openshift-ingress/router-default-944d5c59d-l5g2b"
Sep 16 12:41:08.171792 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:08.171798    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b04ec1dc-5e71-4fed-abc5-6875befad478-service-ca-bundle\") pod \"router-default-944d5c59d-l5g2b\" (UID: \"b04ec1dc-5e71-4fed-abc5-6875befad478\") " pod="openshift-ingress/router-default-944d5c59d-l5g2b"
Sep 16 12:41:08.172532 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:08.172508    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b04ec1dc-5e71-4fed-abc5-6875befad478-service-ca-bundle\") pod \"router-default-944d5c59d-l5g2b\" (UID: \"b04ec1dc-5e71-4fed-abc5-6875befad478\") " pod="openshift-ingress/router-default-944d5c59d-l5g2b"
Sep 16 12:41:08.174558 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:08.174537    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b04ec1dc-5e71-4fed-abc5-6875befad478-metrics-certs\") pod \"router-default-944d5c59d-l5g2b\" (UID: \"b04ec1dc-5e71-4fed-abc5-6875befad478\") " pod="openshift-ingress/router-default-944d5c59d-l5g2b"
Sep 16 12:41:08.182398 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:08.182377    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mlzkb"
Sep 16 12:41:08.390107 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:08.390070    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-944d5c59d-l5g2b"
Sep 16 12:41:08.468595 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:08.468562    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-mlzkb"]
Sep 16 12:41:08.472060 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:41:08.472014    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda583c3d2_5168_4901_89ec_2c64993e60ec.slice/crio-a2372022947f783b426a1b6250d3a03729e148ddb079bdf08ff2a2a49accd539 WatchSource:0}: Error finding container a2372022947f783b426a1b6250d3a03729e148ddb079bdf08ff2a2a49accd539: Status 404 returned error can't find the container with id a2372022947f783b426a1b6250d3a03729e148ddb079bdf08ff2a2a49accd539
Sep 16 12:41:08.624349 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:08.624310    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mlzkb" event={"ID":"a583c3d2-5168-4901-89ec-2c64993e60ec","Type":"ContainerStarted","Data":"a2372022947f783b426a1b6250d3a03729e148ddb079bdf08ff2a2a49accd539"}
Sep 16 12:41:08.681455 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:08.681420    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-944d5c59d-l5g2b"]
Sep 16 12:41:08.685395 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:41:08.685367    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb04ec1dc_5e71_4fed_abc5_6875befad478.slice/crio-b8518e889c5e5439a1a22d894bc7f5f8274654aa3396492b663ecdacbe5d119b WatchSource:0}: Error finding container b8518e889c5e5439a1a22d894bc7f5f8274654aa3396492b663ecdacbe5d119b: Status 404 returned error can't find the container with id b8518e889c5e5439a1a22d894bc7f5f8274654aa3396492b663ecdacbe5d119b
Sep 16 12:41:09.598435 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:09.598403    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ct9vp"
Sep 16 12:41:09.632875 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:09.631971    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-944d5c59d-l5g2b" event={"ID":"b04ec1dc-5e71-4fed-abc5-6875befad478","Type":"ContainerStarted","Data":"3a30dc22bc30490889d63fcf4f365b0cf2558c03bedfd4480d9a8ac23059900a"}
Sep 16 12:41:09.632875 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:09.632049    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-944d5c59d-l5g2b" event={"ID":"b04ec1dc-5e71-4fed-abc5-6875befad478","Type":"ContainerStarted","Data":"b8518e889c5e5439a1a22d894bc7f5f8274654aa3396492b663ecdacbe5d119b"}
Sep 16 12:41:09.656494 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:09.656296    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-944d5c59d-l5g2b" podStartSLOduration=33.656275467 podStartE2EDuration="33.656275467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:41:09.656002076 +0000 UTC m=+176.103234730" watchObservedRunningTime="2026-09-16 12:41:09.656275467 +0000 UTC m=+176.103508107"
Sep 16 12:41:10.391018 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:10.390938    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-944d5c59d-l5g2b"
Sep 16 12:41:10.393990 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:10.393961    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-944d5c59d-l5g2b"
Sep 16 12:41:10.639449 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:10.639344    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mlzkb" event={"ID":"a583c3d2-5168-4901-89ec-2c64993e60ec","Type":"ContainerStarted","Data":"0ecb0fd98ac330bf0c3ca35c28c14d30d659f4bcc1593e0c9f0dda0676170207"}
Sep 16 12:41:10.639984 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:10.639958    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-944d5c59d-l5g2b"
Sep 16 12:41:10.641122 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:10.641070    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-944d5c59d-l5g2b"
Sep 16 12:41:10.669239 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:10.669192    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mlzkb" podStartSLOduration=33.022118061 podStartE2EDuration="34.669174964s" totalImagesPullingTime="1.647056903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:40:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:41:10.667684565 +0000 UTC m=+177.114917206" watchObservedRunningTime="2026-09-16 12:41:10.669174964 +0000 UTC m=+177.116407607"
Sep 16 12:41:10.669976 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:10.669950    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-75qbr"]
Sep 16 12:41:10.673143 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:10.673125    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-75qbr"
Sep 16 12:41:10.675649 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:10.675627    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-t8cjz\""
Sep 16 12:41:10.675762 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:10.675695    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 16 12:41:10.684748 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:10.684725    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-75qbr"]
Sep 16 12:41:10.796820 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:10.796777    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ff18210a-b422-4c2d-bdd8-3332f8c0b78d-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-75qbr\" (UID: \"ff18210a-b422-4c2d-bdd8-3332f8c0b78d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-75qbr"
Sep 16 12:41:10.898218 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:10.898136    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ff18210a-b422-4c2d-bdd8-3332f8c0b78d-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-75qbr\" (UID: \"ff18210a-b422-4c2d-bdd8-3332f8c0b78d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-75qbr"
Sep 16 12:41:10.901321 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:10.901294    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ff18210a-b422-4c2d-bdd8-3332f8c0b78d-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-75qbr\" (UID: \"ff18210a-b422-4c2d-bdd8-3332f8c0b78d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-75qbr"
Sep 16 12:41:10.985262 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:10.985219    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-75qbr"
Sep 16 12:41:11.275863 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:11.275531    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-75qbr"]
Sep 16 12:41:11.279672 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:41:11.279639    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff18210a_b422_4c2d_bdd8_3332f8c0b78d.slice/crio-75e603d5e767fe7f95362c7c6f1ae1c0c5c5824f042757c7fb446bceca54dc0c WatchSource:0}: Error finding container 75e603d5e767fe7f95362c7c6f1ae1c0c5c5824f042757c7fb446bceca54dc0c: Status 404 returned error can't find the container with id 75e603d5e767fe7f95362c7c6f1ae1c0c5c5824f042757c7fb446bceca54dc0c
Sep 16 12:41:11.644583 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:11.644491    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-75qbr" event={"ID":"ff18210a-b422-4c2d-bdd8-3332f8c0b78d","Type":"ContainerStarted","Data":"75e603d5e767fe7f95362c7c6f1ae1c0c5c5824f042757c7fb446bceca54dc0c"}
Sep 16 12:41:12.624909 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:12.624822    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-774d65db68-v4spp"]
Sep 16 12:41:12.628086 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:12.628053    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-774d65db68-v4spp"
Sep 16 12:41:12.631077 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:12.631058    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 12:41:12.631449 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:12.631431    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 12:41:12.632466 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:12.632441    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 12:41:12.632664 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:12.632642    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 12:41:12.632780 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:12.632648    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 12:41:12.632840 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:12.632776    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jjnxd\""
Sep 16 12:41:12.641394 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:12.641370    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-774d65db68-v4spp"]
Sep 16 12:41:12.650593 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:12.650562    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-75qbr" event={"ID":"ff18210a-b422-4c2d-bdd8-3332f8c0b78d","Type":"ContainerStarted","Data":"58d5562fb7055a7eaeaaf23d1c82196440b8afd0b36dd123baee6c37bedf17d5"}
Sep 16 12:41:12.651085 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:12.651042    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-75qbr"
Sep 16 12:41:12.656168 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:12.656148    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-75qbr"
Sep 16 12:41:12.677649 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:12.677595    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-75qbr" podStartSLOduration=1.6692625429999999 podStartE2EDuration="2.677580439s" totalImagesPullingTime="1.008317896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:41:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:41:12.677079597 +0000 UTC m=+179.124312239" watchObservedRunningTime="2026-09-16 12:41:12.677580439 +0000 UTC m=+179.124813082"
Sep 16 12:41:12.717473 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:12.717442    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e614d19-7491-4791-be6f-f6a38547c9c4-console-config\") pod \"console-774d65db68-v4spp\" (UID: \"7e614d19-7491-4791-be6f-f6a38547c9c4\") " pod="openshift-console/console-774d65db68-v4spp"
Sep 16 12:41:12.717626 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:12.717580    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e614d19-7491-4791-be6f-f6a38547c9c4-oauth-serving-cert\") pod \"console-774d65db68-v4spp\" (UID: \"7e614d19-7491-4791-be6f-f6a38547c9c4\") " pod="openshift-console/console-774d65db68-v4spp"
Sep 16 12:41:12.717697 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:12.717648    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e614d19-7491-4791-be6f-f6a38547c9c4-console-oauth-config\") pod \"console-774d65db68-v4spp\" (UID: \"7e614d19-7491-4791-be6f-f6a38547c9c4\") " pod="openshift-console/console-774d65db68-v4spp"
Sep 16 12:41:12.717697 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:12.717675    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4pq8\" (UniqueName: \"kubernetes.io/projected/7e614d19-7491-4791-be6f-f6a38547c9c4-kube-api-access-g4pq8\") pod \"console-774d65db68-v4spp\" (UID: \"7e614d19-7491-4791-be6f-f6a38547c9c4\") " pod="openshift-console/console-774d65db68-v4spp"
Sep 16 12:41:12.717784 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:12.717707    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e614d19-7491-4791-be6f-f6a38547c9c4-console-serving-cert\") pod \"console-774d65db68-v4spp\" (UID: \"7e614d19-7491-4791-be6f-f6a38547c9c4\") " pod="openshift-console/console-774d65db68-v4spp"
Sep 16 12:41:12.717832 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:12.717799    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e614d19-7491-4791-be6f-f6a38547c9c4-service-ca\") pod \"console-774d65db68-v4spp\" (UID: \"7e614d19-7491-4791-be6f-f6a38547c9c4\") " pod="openshift-console/console-774d65db68-v4spp"
Sep 16 12:41:12.819040 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:12.818978    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e614d19-7491-4791-be6f-f6a38547c9c4-oauth-serving-cert\") pod \"console-774d65db68-v4spp\" (UID: \"7e614d19-7491-4791-be6f-f6a38547c9c4\") " pod="openshift-console/console-774d65db68-v4spp"
Sep 16 12:41:12.819217 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:12.819061    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e614d19-7491-4791-be6f-f6a38547c9c4-console-oauth-config\") pod \"console-774d65db68-v4spp\" (UID: \"7e614d19-7491-4791-be6f-f6a38547c9c4\") " pod="openshift-console/console-774d65db68-v4spp"
Sep 16 12:41:12.819217 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:12.819099    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4pq8\" (UniqueName: \"kubernetes.io/projected/7e614d19-7491-4791-be6f-f6a38547c9c4-kube-api-access-g4pq8\") pod \"console-774d65db68-v4spp\" (UID: \"7e614d19-7491-4791-be6f-f6a38547c9c4\") " pod="openshift-console/console-774d65db68-v4spp"
Sep 16 12:41:12.819217 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:12.819127    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e614d19-7491-4791-be6f-f6a38547c9c4-console-serving-cert\") pod \"console-774d65db68-v4spp\" (UID: \"7e614d19-7491-4791-be6f-f6a38547c9c4\") " pod="openshift-console/console-774d65db68-v4spp"
Sep 16 12:41:12.819217 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:12.819160    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e614d19-7491-4791-be6f-f6a38547c9c4-service-ca\") pod \"console-774d65db68-v4spp\" (UID: \"7e614d19-7491-4791-be6f-f6a38547c9c4\") " pod="openshift-console/console-774d65db68-v4spp"
Sep 16 12:41:12.819217 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:12.819218    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e614d19-7491-4791-be6f-f6a38547c9c4-console-config\") pod \"console-774d65db68-v4spp\" (UID: \"7e614d19-7491-4791-be6f-f6a38547c9c4\") " pod="openshift-console/console-774d65db68-v4spp"
Sep 16 12:41:12.819949 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:12.819921    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e614d19-7491-4791-be6f-f6a38547c9c4-service-ca\") pod \"console-774d65db68-v4spp\" (UID: \"7e614d19-7491-4791-be6f-f6a38547c9c4\") " pod="openshift-console/console-774d65db68-v4spp"
Sep 16 12:41:12.820113 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:12.819969    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e614d19-7491-4791-be6f-f6a38547c9c4-console-config\") pod \"console-774d65db68-v4spp\" (UID: \"7e614d19-7491-4791-be6f-f6a38547c9c4\") " pod="openshift-console/console-774d65db68-v4spp"
Sep 16 12:41:12.820310 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:12.820284    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e614d19-7491-4791-be6f-f6a38547c9c4-oauth-serving-cert\") pod \"console-774d65db68-v4spp\" (UID: \"7e614d19-7491-4791-be6f-f6a38547c9c4\") " pod="openshift-console/console-774d65db68-v4spp"
Sep 16 12:41:12.822103 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:12.822080    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e614d19-7491-4791-be6f-f6a38547c9c4-console-oauth-config\") pod \"console-774d65db68-v4spp\" (UID: \"7e614d19-7491-4791-be6f-f6a38547c9c4\") " pod="openshift-console/console-774d65db68-v4spp"
Sep 16 12:41:12.822391 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:12.822375    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e614d19-7491-4791-be6f-f6a38547c9c4-console-serving-cert\") pod \"console-774d65db68-v4spp\" (UID: \"7e614d19-7491-4791-be6f-f6a38547c9c4\") " pod="openshift-console/console-774d65db68-v4spp"
Sep 16 12:41:12.829086 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:12.829065    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4pq8\" (UniqueName: \"kubernetes.io/projected/7e614d19-7491-4791-be6f-f6a38547c9c4-kube-api-access-g4pq8\") pod \"console-774d65db68-v4spp\" (UID: \"7e614d19-7491-4791-be6f-f6a38547c9c4\") " pod="openshift-console/console-774d65db68-v4spp"
Sep 16 12:41:12.938813 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:12.938731    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-774d65db68-v4spp"
Sep 16 12:41:13.021626 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:13.021595    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nps4n\" (UID: \"fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7\") " pod="openshift-insights/insights-runtime-extractor-nps4n"
Sep 16 12:41:13.024474 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:13.024451    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nps4n\" (UID: \"fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7\") " pod="openshift-insights/insights-runtime-extractor-nps4n"
Sep 16 12:41:13.221230 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:13.221159    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-774d65db68-v4spp"]
Sep 16 12:41:13.224329 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:41:13.224302    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e614d19_7491_4791_be6f_f6a38547c9c4.slice/crio-9902b32b0d51aa8199cf17bd35851df852b5fc7a72ab5003200a024ce4290f5f WatchSource:0}: Error finding container 9902b32b0d51aa8199cf17bd35851df852b5fc7a72ab5003200a024ce4290f5f: Status 404 returned error can't find the container with id 9902b32b0d51aa8199cf17bd35851df852b5fc7a72ab5003200a024ce4290f5f
Sep 16 12:41:13.232769 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:13.232748    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nps4n"
Sep 16 12:41:13.514826 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:13.514793    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nps4n"]
Sep 16 12:41:13.518510 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:41:13.518478    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcd51b6e_cc62_4d4d_9ff0_d2fce50916c7.slice/crio-050661bb0183e5955452dd2946a2b618c5e903169b69a0fa004efe963ef6a04d WatchSource:0}: Error finding container 050661bb0183e5955452dd2946a2b618c5e903169b69a0fa004efe963ef6a04d: Status 404 returned error can't find the container with id 050661bb0183e5955452dd2946a2b618c5e903169b69a0fa004efe963ef6a04d
Sep 16 12:41:13.656933 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:13.656894    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nps4n" event={"ID":"fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7","Type":"ContainerStarted","Data":"9c90400de05d6ebdb433d83ed0eb21bc814ae6b8c3313c2b18381174240aa325"}
Sep 16 12:41:13.657408 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:13.656969    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nps4n" event={"ID":"fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7","Type":"ContainerStarted","Data":"050661bb0183e5955452dd2946a2b618c5e903169b69a0fa004efe963ef6a04d"}
Sep 16 12:41:13.659431 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:13.659400    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-774d65db68-v4spp" event={"ID":"7e614d19-7491-4791-be6f-f6a38547c9c4","Type":"ContainerStarted","Data":"9902b32b0d51aa8199cf17bd35851df852b5fc7a72ab5003200a024ce4290f5f"}
Sep 16 12:41:13.719588 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:13.719561    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-q6gv5"]
Sep 16 12:41:13.723187 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:13.723165    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-q6gv5"
Sep 16 12:41:13.725981 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:13.725958    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 16 12:41:13.726086 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:13.725998    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 16 12:41:13.726086 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:13.726009    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-86nsp\""
Sep 16 12:41:13.726086 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:13.726009    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 12:41:13.734438 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:13.734397    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-q6gv5"]
Sep 16 12:41:13.828125 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:13.828016    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mplnz\" (UniqueName: \"kubernetes.io/projected/cb1916e5-42a2-4beb-8deb-0420da03a475-kube-api-access-mplnz\") pod \"prometheus-operator-68bcd7f987-q6gv5\" (UID: \"cb1916e5-42a2-4beb-8deb-0420da03a475\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-q6gv5"
Sep 16 12:41:13.828273 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:13.828125    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb1916e5-42a2-4beb-8deb-0420da03a475-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-q6gv5\" (UID: \"cb1916e5-42a2-4beb-8deb-0420da03a475\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-q6gv5"
Sep 16 12:41:13.828273 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:13.828184    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb1916e5-42a2-4beb-8deb-0420da03a475-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-q6gv5\" (UID: \"cb1916e5-42a2-4beb-8deb-0420da03a475\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-q6gv5"
Sep 16 12:41:13.828273 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:13.828214    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb1916e5-42a2-4beb-8deb-0420da03a475-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-q6gv5\" (UID: \"cb1916e5-42a2-4beb-8deb-0420da03a475\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-q6gv5"
Sep 16 12:41:13.928839 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:13.928798    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb1916e5-42a2-4beb-8deb-0420da03a475-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-q6gv5\" (UID: \"cb1916e5-42a2-4beb-8deb-0420da03a475\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-q6gv5"
Sep 16 12:41:13.929010 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:13.928859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb1916e5-42a2-4beb-8deb-0420da03a475-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-q6gv5\" (UID: \"cb1916e5-42a2-4beb-8deb-0420da03a475\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-q6gv5"
Sep 16 12:41:13.929010 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:13.928937    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mplnz\" (UniqueName: \"kubernetes.io/projected/cb1916e5-42a2-4beb-8deb-0420da03a475-kube-api-access-mplnz\") pod \"prometheus-operator-68bcd7f987-q6gv5\" (UID: \"cb1916e5-42a2-4beb-8deb-0420da03a475\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-q6gv5"
Sep 16 12:41:13.929010 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:13.928970    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb1916e5-42a2-4beb-8deb-0420da03a475-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-q6gv5\" (UID: \"cb1916e5-42a2-4beb-8deb-0420da03a475\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-q6gv5"
Sep 16 12:41:13.929194 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:41:13.929096    2577 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 16 12:41:13.929194 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:41:13.929151    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb1916e5-42a2-4beb-8deb-0420da03a475-prometheus-operator-tls podName:cb1916e5-42a2-4beb-8deb-0420da03a475 nodeName:}" failed. No retries permitted until 2026-09-16 12:41:14.429137459 +0000 UTC m=+180.876370078 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/cb1916e5-42a2-4beb-8deb-0420da03a475-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-q6gv5" (UID: "cb1916e5-42a2-4beb-8deb-0420da03a475") : secret "prometheus-operator-tls" not found
Sep 16 12:41:13.929652 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:13.929623    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb1916e5-42a2-4beb-8deb-0420da03a475-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-q6gv5\" (UID: \"cb1916e5-42a2-4beb-8deb-0420da03a475\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-q6gv5"
Sep 16 12:41:13.931605 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:13.931578    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb1916e5-42a2-4beb-8deb-0420da03a475-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-q6gv5\" (UID: \"cb1916e5-42a2-4beb-8deb-0420da03a475\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-q6gv5"
Sep 16 12:41:13.940732 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:13.940711    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mplnz\" (UniqueName: \"kubernetes.io/projected/cb1916e5-42a2-4beb-8deb-0420da03a475-kube-api-access-mplnz\") pod \"prometheus-operator-68bcd7f987-q6gv5\" (UID: \"cb1916e5-42a2-4beb-8deb-0420da03a475\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-q6gv5"
Sep 16 12:41:14.433729 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:14.433694    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb1916e5-42a2-4beb-8deb-0420da03a475-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-q6gv5\" (UID: \"cb1916e5-42a2-4beb-8deb-0420da03a475\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-q6gv5"
Sep 16 12:41:14.437073 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:14.437020    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb1916e5-42a2-4beb-8deb-0420da03a475-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-q6gv5\" (UID: \"cb1916e5-42a2-4beb-8deb-0420da03a475\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-q6gv5"
Sep 16 12:41:14.643194 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:14.643161    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-86nsp\""
Sep 16 12:41:14.650606 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:14.650580    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-q6gv5"
Sep 16 12:41:14.940812 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:14.940758    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-q6gv5"]
Sep 16 12:41:17.082132 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:17.082094    2577 patch_prober.go:28] interesting pod/image-registry-6fd89fff9d-lkz8m container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 12:41:17.082579 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:17.082153    2577 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m" podUID="70c5a239-65d0-4d57-861b-0ba00290c8bf" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 12:41:18.594667 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:18.594631    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m"
Sep 16 12:41:22.896731 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:41:22.896683    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb1916e5_42a2_4beb_8deb_0420da03a475.slice/crio-96dd95a0f5f23ac83fc56e22d8c9582b5ad86d7e8324d54a4d09b6230d6a7144 WatchSource:0}: Error finding container 96dd95a0f5f23ac83fc56e22d8c9582b5ad86d7e8324d54a4d09b6230d6a7144: Status 404 returned error can't find the container with id 96dd95a0f5f23ac83fc56e22d8c9582b5ad86d7e8324d54a4d09b6230d6a7144
Sep 16 12:41:23.707917 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:23.707830    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-kl8tl" event={"ID":"c41aa457-55d7-4f45-a1b8-f42d4597182c","Type":"ContainerStarted","Data":"f696afc90f30c62af7511179f78fa5713edd3720a15f016066819fde9033d4dd"}
Sep 16 12:41:23.707917 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:23.708350    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-kl8tl"
Sep 16 12:41:23.710996 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:23.710940    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-q6gv5" event={"ID":"cb1916e5-42a2-4beb-8deb-0420da03a475","Type":"ContainerStarted","Data":"96dd95a0f5f23ac83fc56e22d8c9582b5ad86d7e8324d54a4d09b6230d6a7144"}
Sep 16 12:41:23.715070 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:23.715014    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nps4n" event={"ID":"fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7","Type":"ContainerStarted","Data":"6c6a35156fd35ee99831696f9bbfe50f16036cfae144c3e03f8a980a0be9415a"}
Sep 16 12:41:23.718799 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:23.718758    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-774d65db68-v4spp" event={"ID":"7e614d19-7491-4791-be6f-f6a38547c9c4","Type":"ContainerStarted","Data":"1dab522b88337c34e9eaa74f4ffdedeffa61c095e080a947dc873f7c69d74064"}
Sep 16 12:41:23.719308 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:23.719285    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-kl8tl"
Sep 16 12:41:23.733279 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:23.731289    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-kl8tl" podStartSLOduration=1.967779634 podStartE2EDuration="19.731269638s" totalImagesPullingTime="17.763490004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:41:23.730041268 +0000 UTC m=+190.177273901" watchObservedRunningTime="2026-09-16 12:41:23.731269638 +0000 UTC m=+190.178502281"
Sep 16 12:41:23.784247 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:23.784187    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-774d65db68-v4spp" podStartSLOduration=1.983875088 podStartE2EDuration="11.784163649s" totalImagesPullingTime="9.800288561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:41:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:41:23.783608766 +0000 UTC m=+190.230841410" watchObservedRunningTime="2026-09-16 12:41:23.784163649 +0000 UTC m=+190.231396296"
Sep 16 12:41:25.732503 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:25.732160    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-q6gv5" event={"ID":"cb1916e5-42a2-4beb-8deb-0420da03a475","Type":"ContainerStarted","Data":"2b9f877cb73e5cb4405bde1b7827e5ac6104018ee0d340a476e362ebd0e4d313"}
Sep 16 12:41:25.732953 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:25.732611    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-q6gv5" event={"ID":"cb1916e5-42a2-4beb-8deb-0420da03a475","Type":"ContainerStarted","Data":"00b21b4306caeb08d2a4ccc335643b7371d53ae574bfd5a765631a514bb9652c"}
Sep 16 12:41:25.737076 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:25.736580    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nps4n" event={"ID":"fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7","Type":"ContainerStarted","Data":"c09b0271eb7f8f524c4448308bcb4c4b9771861d12c806b691cf028ece65fd40"}
Sep 16 12:41:25.757006 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:25.756933    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nps4n" podStartSLOduration=32.961404315 podStartE2EDuration="44.756912012s" totalImagesPullingTime="11.795507697s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:40:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:41:25.755465433 +0000 UTC m=+192.202698074" watchObservedRunningTime="2026-09-16 12:41:25.756912012 +0000 UTC m=+192.204144655"
Sep 16 12:41:26.761831 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:26.761781    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-q6gv5" podStartSLOduration=11.258439263 podStartE2EDuration="13.761761768s" totalImagesPullingTime="2.503322505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:41:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:41:26.759703081 +0000 UTC m=+193.206935722" watchObservedRunningTime="2026-09-16 12:41:26.761761768 +0000 UTC m=+193.208994410"
Sep 16 12:41:27.164588 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:27.164513    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6fd89fff9d-lkz8m"]
Sep 16 12:41:29.106259 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.106222    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-jvdbf"]
Sep 16 12:41:29.133087 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.133058    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jvdbf"
Sep 16 12:41:29.137068 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.137008    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 16 12:41:29.138105 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.138079    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-jvdbf"]
Sep 16 12:41:29.139413 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.139388    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 16 12:41:29.139582 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.139563    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-fnszz"]
Sep 16 12:41:29.139669 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.139623    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-klg45\""
Sep 16 12:41:29.169303 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.169276    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jbh4p"]
Sep 16 12:41:29.169520 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.169490    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fnszz"
Sep 16 12:41:29.173304 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.173282    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 16 12:41:29.173443 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.173420    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 16 12:41:29.173718 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.173514    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-qmw4m\""
Sep 16 12:41:29.174495 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.174475    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 16 12:41:29.185853 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.185833    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-fnszz"]
Sep 16 12:41:29.186054 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.186039    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jbh4p"
Sep 16 12:41:29.188733 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.188710    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 12:41:29.190214 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.190196    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 12:41:29.190426 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.190406    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 12:41:29.190574 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.190492    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-krqlf\""
Sep 16 12:41:29.266559 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.266522    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d264703a-7eb5-46a8-9dc6-49c3bdfb84e0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jbh4p\" (UID: \"d264703a-7eb5-46a8-9dc6-49c3bdfb84e0\") " pod="openshift-monitoring/node-exporter-jbh4p"
Sep 16 12:41:29.266728 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.266576    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d264703a-7eb5-46a8-9dc6-49c3bdfb84e0-node-exporter-textfile\") pod \"node-exporter-jbh4p\" (UID: \"d264703a-7eb5-46a8-9dc6-49c3bdfb84e0\") " pod="openshift-monitoring/node-exporter-jbh4p"
Sep 16 12:41:29.266728 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.266634    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqsdq\" (UniqueName: \"kubernetes.io/projected/d264703a-7eb5-46a8-9dc6-49c3bdfb84e0-kube-api-access-kqsdq\") pod \"node-exporter-jbh4p\" (UID: \"d264703a-7eb5-46a8-9dc6-49c3bdfb84e0\") " pod="openshift-monitoring/node-exporter-jbh4p"
Sep 16 12:41:29.266728 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.266689    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b2aba6f-e269-465a-9bf6-7d1059876ffb-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-fnszz\" (UID: \"2b2aba6f-e269-465a-9bf6-7d1059876ffb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fnszz"
Sep 16 12:41:29.266728 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.266713    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d264703a-7eb5-46a8-9dc6-49c3bdfb84e0-node-exporter-accelerators-collector-config\") pod \"node-exporter-jbh4p\" (UID: \"d264703a-7eb5-46a8-9dc6-49c3bdfb84e0\") " pod="openshift-monitoring/node-exporter-jbh4p"
Sep 16 12:41:29.266934 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.266739    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d264703a-7eb5-46a8-9dc6-49c3bdfb84e0-root\") pod \"node-exporter-jbh4p\" (UID: \"d264703a-7eb5-46a8-9dc6-49c3bdfb84e0\") " pod="openshift-monitoring/node-exporter-jbh4p"
Sep 16 12:41:29.266934 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.266771    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2b2aba6f-e269-465a-9bf6-7d1059876ffb-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-fnszz\" (UID: \"2b2aba6f-e269-465a-9bf6-7d1059876ffb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fnszz"
Sep 16 12:41:29.266934 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.266806    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aa46cae3-6f61-42f7-89c1-be418493471c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-jvdbf\" (UID: \"aa46cae3-6f61-42f7-89c1-be418493471c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jvdbf"
Sep 16 12:41:29.266934 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.266905    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa46cae3-6f61-42f7-89c1-be418493471c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-jvdbf\" (UID: \"aa46cae3-6f61-42f7-89c1-be418493471c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jvdbf"
Sep 16 12:41:29.267162 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.266940    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2b2aba6f-e269-465a-9bf6-7d1059876ffb-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-fnszz\" (UID: \"2b2aba6f-e269-465a-9bf6-7d1059876ffb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fnszz"
Sep 16 12:41:29.267162 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.266965    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d264703a-7eb5-46a8-9dc6-49c3bdfb84e0-sys\") pod \"node-exporter-jbh4p\" (UID: \"d264703a-7eb5-46a8-9dc6-49c3bdfb84e0\") " pod="openshift-monitoring/node-exporter-jbh4p"
Sep 16 12:41:29.267162 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.267010    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bsjl\" (UniqueName: \"kubernetes.io/projected/2b2aba6f-e269-465a-9bf6-7d1059876ffb-kube-api-access-6bsjl\") pod \"kube-state-metrics-59fbdd77d-fnszz\" (UID: \"2b2aba6f-e269-465a-9bf6-7d1059876ffb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fnszz"
Sep 16 12:41:29.267162 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.267062    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hgcr\" (UniqueName: \"kubernetes.io/projected/aa46cae3-6f61-42f7-89c1-be418493471c-kube-api-access-8hgcr\") pod \"openshift-state-metrics-67dcf8c979-jvdbf\" (UID: \"aa46cae3-6f61-42f7-89c1-be418493471c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jvdbf"
Sep 16 12:41:29.267162 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.267091    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d264703a-7eb5-46a8-9dc6-49c3bdfb84e0-metrics-client-ca\") pod \"node-exporter-jbh4p\" (UID: \"d264703a-7eb5-46a8-9dc6-49c3bdfb84e0\") " pod="openshift-monitoring/node-exporter-jbh4p"
Sep 16 12:41:29.267162 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.267124    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b2aba6f-e269-465a-9bf6-7d1059876ffb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-fnszz\" (UID: \"2b2aba6f-e269-465a-9bf6-7d1059876ffb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fnszz"
Sep 16 12:41:29.267162 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.267150    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2b2aba6f-e269-465a-9bf6-7d1059876ffb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-fnszz\" (UID: \"2b2aba6f-e269-465a-9bf6-7d1059876ffb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fnszz"
Sep 16 12:41:29.267474 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.267183    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa46cae3-6f61-42f7-89c1-be418493471c-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-jvdbf\" (UID: \"aa46cae3-6f61-42f7-89c1-be418493471c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jvdbf"
Sep 16 12:41:29.267474 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.267208    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d264703a-7eb5-46a8-9dc6-49c3bdfb84e0-node-exporter-tls\") pod \"node-exporter-jbh4p\" (UID: \"d264703a-7eb5-46a8-9dc6-49c3bdfb84e0\") " pod="openshift-monitoring/node-exporter-jbh4p"
Sep 16 12:41:29.267474 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.267232    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d264703a-7eb5-46a8-9dc6-49c3bdfb84e0-node-exporter-wtmp\") pod \"node-exporter-jbh4p\" (UID: \"d264703a-7eb5-46a8-9dc6-49c3bdfb84e0\") " pod="openshift-monitoring/node-exporter-jbh4p"
Sep 16 12:41:29.367951 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.367886    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2b2aba6f-e269-465a-9bf6-7d1059876ffb-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-fnszz\" (UID: \"2b2aba6f-e269-465a-9bf6-7d1059876ffb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fnszz"
Sep 16 12:41:29.367951 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.367921    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d264703a-7eb5-46a8-9dc6-49c3bdfb84e0-sys\") pod \"node-exporter-jbh4p\" (UID: \"d264703a-7eb5-46a8-9dc6-49c3bdfb84e0\") " pod="openshift-monitoring/node-exporter-jbh4p"
Sep 16 12:41:29.368230 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.367957    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bsjl\" (UniqueName: \"kubernetes.io/projected/2b2aba6f-e269-465a-9bf6-7d1059876ffb-kube-api-access-6bsjl\") pod \"kube-state-metrics-59fbdd77d-fnszz\" (UID: \"2b2aba6f-e269-465a-9bf6-7d1059876ffb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fnszz"
Sep 16 12:41:29.368230 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.367988    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hgcr\" (UniqueName: \"kubernetes.io/projected/aa46cae3-6f61-42f7-89c1-be418493471c-kube-api-access-8hgcr\") pod \"openshift-state-metrics-67dcf8c979-jvdbf\" (UID: \"aa46cae3-6f61-42f7-89c1-be418493471c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jvdbf"
Sep 16 12:41:29.368230 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.368018    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d264703a-7eb5-46a8-9dc6-49c3bdfb84e0-metrics-client-ca\") pod \"node-exporter-jbh4p\" (UID: \"d264703a-7eb5-46a8-9dc6-49c3bdfb84e0\") " pod="openshift-monitoring/node-exporter-jbh4p"
Sep 16 12:41:29.368230 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.368053    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d264703a-7eb5-46a8-9dc6-49c3bdfb84e0-sys\") pod \"node-exporter-jbh4p\" (UID: \"d264703a-7eb5-46a8-9dc6-49c3bdfb84e0\") " pod="openshift-monitoring/node-exporter-jbh4p"
Sep 16 12:41:29.368230 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:41:29.368094    2577 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 16 12:41:29.368230 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:41:29.368177    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b2aba6f-e269-465a-9bf6-7d1059876ffb-kube-state-metrics-tls podName:2b2aba6f-e269-465a-9bf6-7d1059876ffb nodeName:}" failed. No retries permitted until 2026-09-16 12:41:29.868156649 +0000 UTC m=+196.315389270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/2b2aba6f-e269-465a-9bf6-7d1059876ffb-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-fnszz" (UID: "2b2aba6f-e269-465a-9bf6-7d1059876ffb") : secret "kube-state-metrics-tls" not found
Sep 16 12:41:29.368230 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.368213    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b2aba6f-e269-465a-9bf6-7d1059876ffb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-fnszz\" (UID: \"2b2aba6f-e269-465a-9bf6-7d1059876ffb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fnszz"
Sep 16 12:41:29.368641 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.368610    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2b2aba6f-e269-465a-9bf6-7d1059876ffb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-fnszz\" (UID: \"2b2aba6f-e269-465a-9bf6-7d1059876ffb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fnszz"
Sep 16 12:41:29.368691 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.368662    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d264703a-7eb5-46a8-9dc6-49c3bdfb84e0-metrics-client-ca\") pod \"node-exporter-jbh4p\" (UID: \"d264703a-7eb5-46a8-9dc6-49c3bdfb84e0\") " pod="openshift-monitoring/node-exporter-jbh4p"
Sep 16 12:41:29.368691 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.368668    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa46cae3-6f61-42f7-89c1-be418493471c-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-jvdbf\" (UID: \"aa46cae3-6f61-42f7-89c1-be418493471c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jvdbf"
Sep 16 12:41:29.368775 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.368693    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d264703a-7eb5-46a8-9dc6-49c3bdfb84e0-node-exporter-tls\") pod \"node-exporter-jbh4p\" (UID: \"d264703a-7eb5-46a8-9dc6-49c3bdfb84e0\") " pod="openshift-monitoring/node-exporter-jbh4p"
Sep 16 12:41:29.368775 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.368717    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d264703a-7eb5-46a8-9dc6-49c3bdfb84e0-node-exporter-wtmp\") pod \"node-exporter-jbh4p\" (UID: \"d264703a-7eb5-46a8-9dc6-49c3bdfb84e0\") " pod="openshift-monitoring/node-exporter-jbh4p"
Sep 16 12:41:29.368775 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.368747    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d264703a-7eb5-46a8-9dc6-49c3bdfb84e0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jbh4p\" (UID: \"d264703a-7eb5-46a8-9dc6-49c3bdfb84e0\") " pod="openshift-monitoring/node-exporter-jbh4p"
Sep 16 12:41:29.368907 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.368795    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d264703a-7eb5-46a8-9dc6-49c3bdfb84e0-node-exporter-textfile\") pod \"node-exporter-jbh4p\" (UID: \"d264703a-7eb5-46a8-9dc6-49c3bdfb84e0\") " pod="openshift-monitoring/node-exporter-jbh4p"
Sep 16 12:41:29.368907 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.368821    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqsdq\" (UniqueName: \"kubernetes.io/projected/d264703a-7eb5-46a8-9dc6-49c3bdfb84e0-kube-api-access-kqsdq\") pod \"node-exporter-jbh4p\" (UID: \"d264703a-7eb5-46a8-9dc6-49c3bdfb84e0\") " pod="openshift-monitoring/node-exporter-jbh4p"
Sep 16 12:41:29.368907 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.368859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b2aba6f-e269-465a-9bf6-7d1059876ffb-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-fnszz\" (UID: \"2b2aba6f-e269-465a-9bf6-7d1059876ffb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fnszz"
Sep 16 12:41:29.368907 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.368865    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d264703a-7eb5-46a8-9dc6-49c3bdfb84e0-node-exporter-wtmp\") pod \"node-exporter-jbh4p\" (UID: \"d264703a-7eb5-46a8-9dc6-49c3bdfb84e0\") " pod="openshift-monitoring/node-exporter-jbh4p"
Sep 16 12:41:29.368907 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.368884    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d264703a-7eb5-46a8-9dc6-49c3bdfb84e0-node-exporter-accelerators-collector-config\") pod \"node-exporter-jbh4p\" (UID: \"d264703a-7eb5-46a8-9dc6-49c3bdfb84e0\") " pod="openshift-monitoring/node-exporter-jbh4p"
Sep 16 12:41:29.369176 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.368910    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d264703a-7eb5-46a8-9dc6-49c3bdfb84e0-root\") pod \"node-exporter-jbh4p\" (UID: \"d264703a-7eb5-46a8-9dc6-49c3bdfb84e0\") " pod="openshift-monitoring/node-exporter-jbh4p"
Sep 16 12:41:29.369176 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.368940    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2b2aba6f-e269-465a-9bf6-7d1059876ffb-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-fnszz\" (UID: \"2b2aba6f-e269-465a-9bf6-7d1059876ffb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fnszz"
Sep 16 12:41:29.369176 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.368970    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aa46cae3-6f61-42f7-89c1-be418493471c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-jvdbf\" (UID: \"aa46cae3-6f61-42f7-89c1-be418493471c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jvdbf"
Sep 16 12:41:29.369176 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.369052    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa46cae3-6f61-42f7-89c1-be418493471c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-jvdbf\" (UID: \"aa46cae3-6f61-42f7-89c1-be418493471c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jvdbf"
Sep 16 12:41:29.369176 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.369096    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d264703a-7eb5-46a8-9dc6-49c3bdfb84e0-node-exporter-textfile\") pod \"node-exporter-jbh4p\" (UID: \"d264703a-7eb5-46a8-9dc6-49c3bdfb84e0\") " pod="openshift-monitoring/node-exporter-jbh4p"
Sep 16 12:41:29.369176 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.369123    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2b2aba6f-e269-465a-9bf6-7d1059876ffb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-fnszz\" (UID: \"2b2aba6f-e269-465a-9bf6-7d1059876ffb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fnszz"
Sep 16 12:41:29.369470 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.369188    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d264703a-7eb5-46a8-9dc6-49c3bdfb84e0-root\") pod \"node-exporter-jbh4p\" (UID: \"d264703a-7eb5-46a8-9dc6-49c3bdfb84e0\") " pod="openshift-monitoring/node-exporter-jbh4p"
Sep 16 12:41:29.369470 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.369382    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa46cae3-6f61-42f7-89c1-be418493471c-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-jvdbf\" (UID: \"aa46cae3-6f61-42f7-89c1-be418493471c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jvdbf"
Sep 16 12:41:29.369576 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.369527    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2b2aba6f-e269-465a-9bf6-7d1059876ffb-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-fnszz\" (UID: \"2b2aba6f-e269-465a-9bf6-7d1059876ffb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fnszz"
Sep 16 12:41:29.369753 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.369652    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d264703a-7eb5-46a8-9dc6-49c3bdfb84e0-node-exporter-accelerators-collector-config\") pod \"node-exporter-jbh4p\" (UID: \"d264703a-7eb5-46a8-9dc6-49c3bdfb84e0\") " pod="openshift-monitoring/node-exporter-jbh4p"
Sep 16 12:41:29.370224 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.369897    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b2aba6f-e269-465a-9bf6-7d1059876ffb-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-fnszz\" (UID: \"2b2aba6f-e269-465a-9bf6-7d1059876ffb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fnszz"
Sep 16 12:41:29.372064 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.372019    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aa46cae3-6f61-42f7-89c1-be418493471c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-jvdbf\" (UID: \"aa46cae3-6f61-42f7-89c1-be418493471c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jvdbf"
Sep 16 12:41:29.372219 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.372195    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b2aba6f-e269-465a-9bf6-7d1059876ffb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-fnszz\" (UID: \"2b2aba6f-e269-465a-9bf6-7d1059876ffb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fnszz"
Sep 16 12:41:29.372605 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.372564    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d264703a-7eb5-46a8-9dc6-49c3bdfb84e0-node-exporter-tls\") pod \"node-exporter-jbh4p\" (UID: \"d264703a-7eb5-46a8-9dc6-49c3bdfb84e0\") " pod="openshift-monitoring/node-exporter-jbh4p"
Sep 16 12:41:29.372709 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.372671    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d264703a-7eb5-46a8-9dc6-49c3bdfb84e0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jbh4p\" (UID: \"d264703a-7eb5-46a8-9dc6-49c3bdfb84e0\") " pod="openshift-monitoring/node-exporter-jbh4p"
Sep 16 12:41:29.373878 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.373850    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa46cae3-6f61-42f7-89c1-be418493471c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-jvdbf\" (UID: \"aa46cae3-6f61-42f7-89c1-be418493471c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jvdbf"
Sep 16 12:41:29.378349 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.378330    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bsjl\" (UniqueName: \"kubernetes.io/projected/2b2aba6f-e269-465a-9bf6-7d1059876ffb-kube-api-access-6bsjl\") pod \"kube-state-metrics-59fbdd77d-fnszz\" (UID: \"2b2aba6f-e269-465a-9bf6-7d1059876ffb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fnszz"
Sep 16 12:41:29.378817 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.378795    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hgcr\" (UniqueName: \"kubernetes.io/projected/aa46cae3-6f61-42f7-89c1-be418493471c-kube-api-access-8hgcr\") pod \"openshift-state-metrics-67dcf8c979-jvdbf\" (UID: \"aa46cae3-6f61-42f7-89c1-be418493471c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jvdbf"
Sep 16 12:41:29.379251 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.379230    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqsdq\" (UniqueName: \"kubernetes.io/projected/d264703a-7eb5-46a8-9dc6-49c3bdfb84e0-kube-api-access-kqsdq\") pod \"node-exporter-jbh4p\" (UID: \"d264703a-7eb5-46a8-9dc6-49c3bdfb84e0\") " pod="openshift-monitoring/node-exporter-jbh4p"
Sep 16 12:41:29.446190 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.446156    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jvdbf"
Sep 16 12:41:29.497122 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.497093    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jbh4p"
Sep 16 12:41:29.657659 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.657574    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-774d65db68-v4spp"]
Sep 16 12:41:29.754768 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.754727    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jbh4p" event={"ID":"d264703a-7eb5-46a8-9dc6-49c3bdfb84e0","Type":"ContainerStarted","Data":"1cd95231a2023977111d2e8e2e7a544e99ea51a052edcb68266527f8fb3d9f06"}
Sep 16 12:41:29.761495 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.761467    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-jvdbf"]
Sep 16 12:41:29.768844 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:41:29.767596    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa46cae3_6f61_42f7_89c1_be418493471c.slice/crio-cbc0421bff8bed112bd2c23445e47cf83bc294e95999d758bfcec747542a7a13 WatchSource:0}: Error finding container cbc0421bff8bed112bd2c23445e47cf83bc294e95999d758bfcec747542a7a13: Status 404 returned error can't find the container with id cbc0421bff8bed112bd2c23445e47cf83bc294e95999d758bfcec747542a7a13
Sep 16 12:41:29.872780 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.872751    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2b2aba6f-e269-465a-9bf6-7d1059876ffb-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-fnszz\" (UID: \"2b2aba6f-e269-465a-9bf6-7d1059876ffb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fnszz"
Sep 16 12:41:29.876212 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:29.876133    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2b2aba6f-e269-465a-9bf6-7d1059876ffb-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-fnszz\" (UID: \"2b2aba6f-e269-465a-9bf6-7d1059876ffb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fnszz"
Sep 16 12:41:30.082582 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:30.082335    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fnszz"
Sep 16 12:41:30.400686 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:30.400589    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-fnszz"]
Sep 16 12:41:30.407131 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:41:30.406912    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b2aba6f_e269_465a_9bf6_7d1059876ffb.slice/crio-85f46d4a64bd2d1d74f721a1f4230d8a134d00db88008deb06411d7226dc1100 WatchSource:0}: Error finding container 85f46d4a64bd2d1d74f721a1f4230d8a134d00db88008deb06411d7226dc1100: Status 404 returned error can't find the container with id 85f46d4a64bd2d1d74f721a1f4230d8a134d00db88008deb06411d7226dc1100
Sep 16 12:41:30.762106 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:30.762065    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jvdbf" event={"ID":"aa46cae3-6f61-42f7-89c1-be418493471c","Type":"ContainerStarted","Data":"c3d527a50aab2847ebbdfe0238609e6d05880c716c42849dcc1f2b9caaed790e"}
Sep 16 12:41:30.762350 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:30.762138    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jvdbf" event={"ID":"aa46cae3-6f61-42f7-89c1-be418493471c","Type":"ContainerStarted","Data":"d9e0e819140cfb0ca5edcaac1a2839cd59b32a80ca794e66bb01c4e731693a17"}
Sep 16 12:41:30.762350 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:30.762164    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jvdbf" event={"ID":"aa46cae3-6f61-42f7-89c1-be418493471c","Type":"ContainerStarted","Data":"cbc0421bff8bed112bd2c23445e47cf83bc294e95999d758bfcec747542a7a13"}
Sep 16 12:41:30.763505 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:30.763475    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fnszz" event={"ID":"2b2aba6f-e269-465a-9bf6-7d1059876ffb","Type":"ContainerStarted","Data":"85f46d4a64bd2d1d74f721a1f4230d8a134d00db88008deb06411d7226dc1100"}
Sep 16 12:41:31.772095 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:31.772058    2577 generic.go:356] "Generic (PLEG): container finished" podID="d264703a-7eb5-46a8-9dc6-49c3bdfb84e0" containerID="b9df780a57e2e90b774e4df158935f116f2ea3e824b42c20f0b99fbbff4dcf3c" exitCode=0
Sep 16 12:41:31.772588 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:31.772181    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jbh4p" event={"ID":"d264703a-7eb5-46a8-9dc6-49c3bdfb84e0","Type":"ContainerDied","Data":"b9df780a57e2e90b774e4df158935f116f2ea3e824b42c20f0b99fbbff4dcf3c"}
Sep 16 12:41:32.778865 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:32.778834    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jbh4p" event={"ID":"d264703a-7eb5-46a8-9dc6-49c3bdfb84e0","Type":"ContainerStarted","Data":"910d558d51a748eccfe57f6cb1859d9a8bec6b0da5453da59c9c3a3460d1c84a"}
Sep 16 12:41:32.779369 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:32.778889    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jbh4p" event={"ID":"d264703a-7eb5-46a8-9dc6-49c3bdfb84e0","Type":"ContainerStarted","Data":"6de2dbb2294d998dafd2281a50485d5bbcb00f3dc91982e6965b99eb152944fa"}
Sep 16 12:41:32.808110 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:32.808043    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jbh4p" podStartSLOduration=2.093947358 podStartE2EDuration="3.808005348s" totalImagesPullingTime="1.617723219s" totalInitContainerRuntime="96.334771ms" isStatefulPod=true podCreationTimestamp="2026-09-16 12:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:41:32.804988285 +0000 UTC m=+199.252220927" watchObservedRunningTime="2026-09-16 12:41:32.808005348 +0000 UTC m=+199.255237993"
Sep 16 12:41:32.939867 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:32.939826    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-774d65db68-v4spp"
Sep 16 12:41:33.786318 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:33.786285    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fnszz" event={"ID":"2b2aba6f-e269-465a-9bf6-7d1059876ffb","Type":"ContainerStarted","Data":"beb82b329785207a96d566e09f6e830657b65175d245b8c32436cc8b53a5a1c7"}
Sep 16 12:41:33.786708 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:33.786360    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fnszz" event={"ID":"2b2aba6f-e269-465a-9bf6-7d1059876ffb","Type":"ContainerStarted","Data":"1a79d47d34a6beefa0c9bd493f68ab1b44c23811c3c777a3cb8533fedbb1c333"}
Sep 16 12:41:33.789061 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:33.789018    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jvdbf" event={"ID":"aa46cae3-6f61-42f7-89c1-be418493471c","Type":"ContainerStarted","Data":"899d70d0212ae4aab689ea198178ddeddfd356fdf5a428be572b3d0ec752c708"}
Sep 16 12:41:33.824940 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:33.824897    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jvdbf" podStartSLOduration=1.743426084 podStartE2EDuration="4.824882607s" totalImagesPullingTime="3.081456523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:41:33.822970523 +0000 UTC m=+200.270203165" watchObservedRunningTime="2026-09-16 12:41:33.824882607 +0000 UTC m=+200.272115249"
Sep 16 12:41:34.795448 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:34.795410    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fnszz" event={"ID":"2b2aba6f-e269-465a-9bf6-7d1059876ffb","Type":"ContainerStarted","Data":"f7576e0908e42acd420522732d49d66a82f724d07101d157da217bb1ec81dc5f"}
Sep 16 12:41:34.821642 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:34.821594    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fnszz" podStartSLOduration=2.722670504 podStartE2EDuration="5.821579272s" totalImagesPullingTime="3.098908768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:41:34.820097606 +0000 UTC m=+201.267330249" watchObservedRunningTime="2026-09-16 12:41:34.821579272 +0000 UTC m=+201.268811914"
Sep 16 12:41:52.186967 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:52.186929    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m" podUID="70c5a239-65d0-4d57-861b-0ba00290c8bf" containerName="registry" containerID="cri-o://7b483a1bc659962bb06f6091c8ccd8ccfeb0bef1ee390927aa2db7f6f258e3d3" gracePeriod=30
Sep 16 12:41:52.458281 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:52.458223    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m"
Sep 16 12:41:52.559061 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:52.559011    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-registry-tls\") pod \"70c5a239-65d0-4d57-861b-0ba00290c8bf\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") "
Sep 16 12:41:52.559224 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:52.559078    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/70c5a239-65d0-4d57-861b-0ba00290c8bf-image-registry-private-configuration\") pod \"70c5a239-65d0-4d57-861b-0ba00290c8bf\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") "
Sep 16 12:41:52.559224 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:52.559099    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/70c5a239-65d0-4d57-861b-0ba00290c8bf-installation-pull-secrets\") pod \"70c5a239-65d0-4d57-861b-0ba00290c8bf\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") "
Sep 16 12:41:52.559224 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:52.559120    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-bound-sa-token\") pod \"70c5a239-65d0-4d57-861b-0ba00290c8bf\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") "
Sep 16 12:41:52.559224 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:52.559140    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/70c5a239-65d0-4d57-861b-0ba00290c8bf-ca-trust-extracted\") pod \"70c5a239-65d0-4d57-861b-0ba00290c8bf\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") "
Sep 16 12:41:52.559224 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:52.559165    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/70c5a239-65d0-4d57-861b-0ba00290c8bf-trusted-ca\") pod \"70c5a239-65d0-4d57-861b-0ba00290c8bf\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") "
Sep 16 12:41:52.559224 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:52.559182    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/70c5a239-65d0-4d57-861b-0ba00290c8bf-registry-certificates\") pod \"70c5a239-65d0-4d57-861b-0ba00290c8bf\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") "
Sep 16 12:41:52.559224 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:52.559210    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7nv6\" (UniqueName: \"kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-kube-api-access-k7nv6\") pod \"70c5a239-65d0-4d57-861b-0ba00290c8bf\" (UID: \"70c5a239-65d0-4d57-861b-0ba00290c8bf\") "
Sep 16 12:41:52.559944 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:52.559726    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70c5a239-65d0-4d57-861b-0ba00290c8bf-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "70c5a239-65d0-4d57-861b-0ba00290c8bf" (UID: "70c5a239-65d0-4d57-861b-0ba00290c8bf"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:41:52.559944 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:52.559751    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70c5a239-65d0-4d57-861b-0ba00290c8bf-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "70c5a239-65d0-4d57-861b-0ba00290c8bf" (UID: "70c5a239-65d0-4d57-861b-0ba00290c8bf"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:41:52.561709 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:52.561655    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70c5a239-65d0-4d57-861b-0ba00290c8bf-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "70c5a239-65d0-4d57-861b-0ba00290c8bf" (UID: "70c5a239-65d0-4d57-861b-0ba00290c8bf"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:41:52.561826 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:52.561716    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "70c5a239-65d0-4d57-861b-0ba00290c8bf" (UID: "70c5a239-65d0-4d57-861b-0ba00290c8bf"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:41:52.561826 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:52.561722    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "70c5a239-65d0-4d57-861b-0ba00290c8bf" (UID: "70c5a239-65d0-4d57-861b-0ba00290c8bf"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:41:52.562201 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:52.562177    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-kube-api-access-k7nv6" (OuterVolumeSpecName: "kube-api-access-k7nv6") pod "70c5a239-65d0-4d57-861b-0ba00290c8bf" (UID: "70c5a239-65d0-4d57-861b-0ba00290c8bf"). InnerVolumeSpecName "kube-api-access-k7nv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:41:52.562369 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:52.562351    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70c5a239-65d0-4d57-861b-0ba00290c8bf-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "70c5a239-65d0-4d57-861b-0ba00290c8bf" (UID: "70c5a239-65d0-4d57-861b-0ba00290c8bf"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:41:52.567848 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:52.567821    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c5a239-65d0-4d57-861b-0ba00290c8bf-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "70c5a239-65d0-4d57-861b-0ba00290c8bf" (UID: "70c5a239-65d0-4d57-861b-0ba00290c8bf"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:41:52.660904 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:52.660872    2577 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/70c5a239-65d0-4d57-861b-0ba00290c8bf-image-registry-private-configuration\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:41:52.660904 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:52.660898    2577 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/70c5a239-65d0-4d57-861b-0ba00290c8bf-installation-pull-secrets\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:41:52.660904 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:52.660910    2577 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-bound-sa-token\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:41:52.661164 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:52.660922    2577 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/70c5a239-65d0-4d57-861b-0ba00290c8bf-ca-trust-extracted\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:41:52.661164 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:52.660933    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/70c5a239-65d0-4d57-861b-0ba00290c8bf-trusted-ca\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:41:52.661164 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:52.660946    2577 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/70c5a239-65d0-4d57-861b-0ba00290c8bf-registry-certificates\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:41:52.661164 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:52.660956    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7nv6\" (UniqueName: \"kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-kube-api-access-k7nv6\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:41:52.661164 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:52.660967    2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70c5a239-65d0-4d57-861b-0ba00290c8bf-registry-tls\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:41:52.865090 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:52.865054    2577 generic.go:356] "Generic (PLEG): container finished" podID="70c5a239-65d0-4d57-861b-0ba00290c8bf" containerID="7b483a1bc659962bb06f6091c8ccd8ccfeb0bef1ee390927aa2db7f6f258e3d3" exitCode=0
Sep 16 12:41:52.865252 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:52.865102    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m"
Sep 16 12:41:52.865252 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:52.865116    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m" event={"ID":"70c5a239-65d0-4d57-861b-0ba00290c8bf","Type":"ContainerDied","Data":"7b483a1bc659962bb06f6091c8ccd8ccfeb0bef1ee390927aa2db7f6f258e3d3"}
Sep 16 12:41:52.865252 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:52.865173    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fd89fff9d-lkz8m" event={"ID":"70c5a239-65d0-4d57-861b-0ba00290c8bf","Type":"ContainerDied","Data":"401d5dede241162cf747cbf5c553bce3334bcb9709444c167bfe1e452d771067"}
Sep 16 12:41:52.865252 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:52.865193    2577 scope.go:117] "RemoveContainer" containerID="7b483a1bc659962bb06f6091c8ccd8ccfeb0bef1ee390927aa2db7f6f258e3d3"
Sep 16 12:41:52.881499 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:52.881481    2577 scope.go:117] "RemoveContainer" containerID="7b483a1bc659962bb06f6091c8ccd8ccfeb0bef1ee390927aa2db7f6f258e3d3"
Sep 16 12:41:52.881818 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:41:52.881799    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b483a1bc659962bb06f6091c8ccd8ccfeb0bef1ee390927aa2db7f6f258e3d3\": container with ID starting with 7b483a1bc659962bb06f6091c8ccd8ccfeb0bef1ee390927aa2db7f6f258e3d3 not found: ID does not exist" containerID="7b483a1bc659962bb06f6091c8ccd8ccfeb0bef1ee390927aa2db7f6f258e3d3"
Sep 16 12:41:52.881868 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:52.881826    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b483a1bc659962bb06f6091c8ccd8ccfeb0bef1ee390927aa2db7f6f258e3d3"} err="failed to get container status \"7b483a1bc659962bb06f6091c8ccd8ccfeb0bef1ee390927aa2db7f6f258e3d3\": rpc error: code = NotFound desc = could not find container \"7b483a1bc659962bb06f6091c8ccd8ccfeb0bef1ee390927aa2db7f6f258e3d3\": container with ID starting with 7b483a1bc659962bb06f6091c8ccd8ccfeb0bef1ee390927aa2db7f6f258e3d3 not found: ID does not exist"
Sep 16 12:41:52.887297 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:52.887276    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6fd89fff9d-lkz8m"]
Sep 16 12:41:52.892816 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:52.892796    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6fd89fff9d-lkz8m"]
Sep 16 12:41:54.053846 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:54.053814    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70c5a239-65d0-4d57-861b-0ba00290c8bf" path="/var/lib/kubelet/pods/70c5a239-65d0-4d57-861b-0ba00290c8bf/volumes"
Sep 16 12:41:54.680441 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:54.680401    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-774d65db68-v4spp" podUID="7e614d19-7491-4791-be6f-f6a38547c9c4" containerName="console" containerID="cri-o://1dab522b88337c34e9eaa74f4ffdedeffa61c095e080a947dc873f7c69d74064" gracePeriod=15
Sep 16 12:41:54.877564 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:54.877530    2577 generic.go:356] "Generic (PLEG): container finished" podID="7e614d19-7491-4791-be6f-f6a38547c9c4" containerID="1dab522b88337c34e9eaa74f4ffdedeffa61c095e080a947dc873f7c69d74064" exitCode=0
Sep 16 12:41:54.877681 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:54.877582    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-774d65db68-v4spp" event={"ID":"7e614d19-7491-4791-be6f-f6a38547c9c4","Type":"ContainerDied","Data":"1dab522b88337c34e9eaa74f4ffdedeffa61c095e080a947dc873f7c69d74064"}
Sep 16 12:41:54.962553 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:54.962491    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-774d65db68-v4spp"
Sep 16 12:41:54.977803 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:54.977779    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e614d19-7491-4791-be6f-f6a38547c9c4-oauth-serving-cert\") pod \"7e614d19-7491-4791-be6f-f6a38547c9c4\" (UID: \"7e614d19-7491-4791-be6f-f6a38547c9c4\") "
Sep 16 12:41:54.977899 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:54.977823    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e614d19-7491-4791-be6f-f6a38547c9c4-service-ca\") pod \"7e614d19-7491-4791-be6f-f6a38547c9c4\" (UID: \"7e614d19-7491-4791-be6f-f6a38547c9c4\") "
Sep 16 12:41:54.977899 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:54.977875    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e614d19-7491-4791-be6f-f6a38547c9c4-console-serving-cert\") pod \"7e614d19-7491-4791-be6f-f6a38547c9c4\" (UID: \"7e614d19-7491-4791-be6f-f6a38547c9c4\") "
Sep 16 12:41:54.977899 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:54.977894    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e614d19-7491-4791-be6f-f6a38547c9c4-console-oauth-config\") pod \"7e614d19-7491-4791-be6f-f6a38547c9c4\" (UID: \"7e614d19-7491-4791-be6f-f6a38547c9c4\") "
Sep 16 12:41:54.978068 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:54.977915    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4pq8\" (UniqueName: \"kubernetes.io/projected/7e614d19-7491-4791-be6f-f6a38547c9c4-kube-api-access-g4pq8\") pod \"7e614d19-7491-4791-be6f-f6a38547c9c4\" (UID: \"7e614d19-7491-4791-be6f-f6a38547c9c4\") "
Sep 16 12:41:54.978068 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:54.977942    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e614d19-7491-4791-be6f-f6a38547c9c4-console-config\") pod \"7e614d19-7491-4791-be6f-f6a38547c9c4\" (UID: \"7e614d19-7491-4791-be6f-f6a38547c9c4\") "
Sep 16 12:41:54.978244 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:54.978220    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e614d19-7491-4791-be6f-f6a38547c9c4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7e614d19-7491-4791-be6f-f6a38547c9c4" (UID: "7e614d19-7491-4791-be6f-f6a38547c9c4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:41:54.978409 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:54.978376    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e614d19-7491-4791-be6f-f6a38547c9c4-service-ca" (OuterVolumeSpecName: "service-ca") pod "7e614d19-7491-4791-be6f-f6a38547c9c4" (UID: "7e614d19-7491-4791-be6f-f6a38547c9c4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:41:54.978488 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:54.978429    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e614d19-7491-4791-be6f-f6a38547c9c4-console-config" (OuterVolumeSpecName: "console-config") pod "7e614d19-7491-4791-be6f-f6a38547c9c4" (UID: "7e614d19-7491-4791-be6f-f6a38547c9c4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:41:54.980400 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:54.980376    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e614d19-7491-4791-be6f-f6a38547c9c4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7e614d19-7491-4791-be6f-f6a38547c9c4" (UID: "7e614d19-7491-4791-be6f-f6a38547c9c4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:41:54.980771 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:54.980745    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e614d19-7491-4791-be6f-f6a38547c9c4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7e614d19-7491-4791-be6f-f6a38547c9c4" (UID: "7e614d19-7491-4791-be6f-f6a38547c9c4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:41:54.980771 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:54.980758    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e614d19-7491-4791-be6f-f6a38547c9c4-kube-api-access-g4pq8" (OuterVolumeSpecName: "kube-api-access-g4pq8") pod "7e614d19-7491-4791-be6f-f6a38547c9c4" (UID: "7e614d19-7491-4791-be6f-f6a38547c9c4"). InnerVolumeSpecName "kube-api-access-g4pq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:41:55.079172 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:55.079141    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e614d19-7491-4791-be6f-f6a38547c9c4-console-config\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:41:55.079172 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:55.079169    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e614d19-7491-4791-be6f-f6a38547c9c4-oauth-serving-cert\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:41:55.079172 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:55.079180    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e614d19-7491-4791-be6f-f6a38547c9c4-service-ca\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:41:55.079609 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:55.079188    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e614d19-7491-4791-be6f-f6a38547c9c4-console-serving-cert\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:41:55.079609 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:55.079197    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e614d19-7491-4791-be6f-f6a38547c9c4-console-oauth-config\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:41:55.079609 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:55.079205    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4pq8\" (UniqueName: \"kubernetes.io/projected/7e614d19-7491-4791-be6f-f6a38547c9c4-kube-api-access-g4pq8\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:41:55.882706 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:55.882675    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-774d65db68-v4spp"
Sep 16 12:41:55.882859 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:55.882669    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-774d65db68-v4spp" event={"ID":"7e614d19-7491-4791-be6f-f6a38547c9c4","Type":"ContainerDied","Data":"9902b32b0d51aa8199cf17bd35851df852b5fc7a72ab5003200a024ce4290f5f"}
Sep 16 12:41:55.882859 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:55.882811    2577 scope.go:117] "RemoveContainer" containerID="1dab522b88337c34e9eaa74f4ffdedeffa61c095e080a947dc873f7c69d74064"
Sep 16 12:41:55.903076 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:55.903050    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-774d65db68-v4spp"]
Sep 16 12:41:55.906911 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:55.906880    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-774d65db68-v4spp"]
Sep 16 12:41:56.053768 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:41:56.053732    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e614d19-7491-4791-be6f-f6a38547c9c4" path="/var/lib/kubelet/pods/7e614d19-7491-4791-be6f-f6a38547c9c4/volumes"
Sep 16 12:42:05.922424 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:05.922383    2577 generic.go:356] "Generic (PLEG): container finished" podID="ad668dd0-69c8-435f-9b75-32f037bfdbc1" containerID="6d4bf62262946a247654cbbc0417c5eb5db38c22267ca6750e64e19acdd340b5" exitCode=0
Sep 16 12:42:05.922933 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:05.922434    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5q4bl" event={"ID":"ad668dd0-69c8-435f-9b75-32f037bfdbc1","Type":"ContainerDied","Data":"6d4bf62262946a247654cbbc0417c5eb5db38c22267ca6750e64e19acdd340b5"}
Sep 16 12:42:05.922933 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:05.922804    2577 scope.go:117] "RemoveContainer" containerID="6d4bf62262946a247654cbbc0417c5eb5db38c22267ca6750e64e19acdd340b5"
Sep 16 12:42:06.928141 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:06.928106    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5q4bl" event={"ID":"ad668dd0-69c8-435f-9b75-32f037bfdbc1","Type":"ContainerStarted","Data":"6467179af4082ed50b9838e972a6dcc000b5442f947e7eda7f53b15506594686"}
Sep 16 12:42:25.824249 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:25.824212    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5-metrics-certs\") pod \"network-metrics-daemon-tmfpk\" (UID: \"aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5\") " pod="openshift-multus/network-metrics-daemon-tmfpk"
Sep 16 12:42:25.828070 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:25.828015    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5-metrics-certs\") pod \"network-metrics-daemon-tmfpk\" (UID: \"aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5\") " pod="openshift-multus/network-metrics-daemon-tmfpk"
Sep 16 12:42:25.852491 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:25.852460    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-b66st\""
Sep 16 12:42:25.859871 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:25.859845    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmfpk"
Sep 16 12:42:26.153625 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:26.153500    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tmfpk"]
Sep 16 12:42:26.161417 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:42:26.161391    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa8d4043_5a04_4558_b0a7_d99b5e5c1ca5.slice/crio-aed986a3a91dfca62409925cfaebac2cf4c3bc2aa6803d4c7682b41f7df975e4 WatchSource:0}: Error finding container aed986a3a91dfca62409925cfaebac2cf4c3bc2aa6803d4c7682b41f7df975e4: Status 404 returned error can't find the container with id aed986a3a91dfca62409925cfaebac2cf4c3bc2aa6803d4c7682b41f7df975e4
Sep 16 12:42:26.998916 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:26.998874    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tmfpk" event={"ID":"aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5","Type":"ContainerStarted","Data":"aed986a3a91dfca62409925cfaebac2cf4c3bc2aa6803d4c7682b41f7df975e4"}
Sep 16 12:42:28.003748 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:28.003707    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tmfpk" event={"ID":"aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5","Type":"ContainerStarted","Data":"c2de65bc2dd27f2c0ec47ec118a6fdd71382f374d2a7dc28bb5b91c815480a11"}
Sep 16 12:42:28.004151 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:28.003758    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tmfpk" event={"ID":"aa8d4043-5a04-4558-b0a7-d99b5e5c1ca5","Type":"ContainerStarted","Data":"e09866a410e22e24ecc9e74f996e824ac016aa88dfa640755d2bdb8611af2c96"}
Sep 16 12:42:28.022360 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:28.022316    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tmfpk" podStartSLOduration=252.037506211 podStartE2EDuration="4m13.02230039s" totalImagesPullingTime="984.794179ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:38:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:42:28.02134036 +0000 UTC m=+254.468573003" watchObservedRunningTime="2026-09-16 12:42:28.02230039 +0000 UTC m=+254.469533014"
Sep 16 12:42:53.577825 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.577779    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-8b96c8b44-w7j7n"]
Sep 16 12:42:53.578425 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.578402    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e614d19-7491-4791-be6f-f6a38547c9c4" containerName="console"
Sep 16 12:42:53.578509 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.578429    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e614d19-7491-4791-be6f-f6a38547c9c4" containerName="console"
Sep 16 12:42:53.578509 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.578442    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70c5a239-65d0-4d57-861b-0ba00290c8bf" containerName="registry"
Sep 16 12:42:53.578509 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.578449    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="70c5a239-65d0-4d57-861b-0ba00290c8bf" containerName="registry"
Sep 16 12:42:53.578658 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.578523    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="70c5a239-65d0-4d57-861b-0ba00290c8bf" containerName="registry"
Sep 16 12:42:53.578658 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.578534    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e614d19-7491-4791-be6f-f6a38547c9c4" containerName="console"
Sep 16 12:42:53.581670 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.581649    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-8b96c8b44-w7j7n"
Sep 16 12:42:53.584622 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.584603    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-pvbhj\""
Sep 16 12:42:53.584732 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.584709    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 16 12:42:53.585150 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.585131    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 16 12:42:53.585266 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.585151    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 16 12:42:53.585266 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.585213    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 16 12:42:53.585455 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.585433    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 16 12:42:53.590817 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.590754    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 16 12:42:53.598325 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.598299    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-8b96c8b44-w7j7n"]
Sep 16 12:42:53.733730 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.733694    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f7d083fa-9933-493c-8caa-c1a2dae789e4-secret-telemeter-client\") pod \"telemeter-client-8b96c8b44-w7j7n\" (UID: \"f7d083fa-9933-493c-8caa-c1a2dae789e4\") " pod="openshift-monitoring/telemeter-client-8b96c8b44-w7j7n"
Sep 16 12:42:53.733904 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.733743    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7d083fa-9933-493c-8caa-c1a2dae789e4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8b96c8b44-w7j7n\" (UID: \"f7d083fa-9933-493c-8caa-c1a2dae789e4\") " pod="openshift-monitoring/telemeter-client-8b96c8b44-w7j7n"
Sep 16 12:42:53.733904 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.733776    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f7d083fa-9933-493c-8caa-c1a2dae789e4-federate-client-tls\") pod \"telemeter-client-8b96c8b44-w7j7n\" (UID: \"f7d083fa-9933-493c-8caa-c1a2dae789e4\") " pod="openshift-monitoring/telemeter-client-8b96c8b44-w7j7n"
Sep 16 12:42:53.733904 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.733819    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f7d083fa-9933-493c-8caa-c1a2dae789e4-telemeter-client-tls\") pod \"telemeter-client-8b96c8b44-w7j7n\" (UID: \"f7d083fa-9933-493c-8caa-c1a2dae789e4\") " pod="openshift-monitoring/telemeter-client-8b96c8b44-w7j7n"
Sep 16 12:42:53.733904 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.733842    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7d083fa-9933-493c-8caa-c1a2dae789e4-serving-certs-ca-bundle\") pod \"telemeter-client-8b96c8b44-w7j7n\" (UID: \"f7d083fa-9933-493c-8caa-c1a2dae789e4\") " pod="openshift-monitoring/telemeter-client-8b96c8b44-w7j7n"
Sep 16 12:42:53.733904 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.733868    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7d083fa-9933-493c-8caa-c1a2dae789e4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8b96c8b44-w7j7n\" (UID: \"f7d083fa-9933-493c-8caa-c1a2dae789e4\") " pod="openshift-monitoring/telemeter-client-8b96c8b44-w7j7n"
Sep 16 12:42:53.733904 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.733892    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgbv4\" (UniqueName: \"kubernetes.io/projected/f7d083fa-9933-493c-8caa-c1a2dae789e4-kube-api-access-tgbv4\") pod \"telemeter-client-8b96c8b44-w7j7n\" (UID: \"f7d083fa-9933-493c-8caa-c1a2dae789e4\") " pod="openshift-monitoring/telemeter-client-8b96c8b44-w7j7n"
Sep 16 12:42:53.734144 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.733931    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7d083fa-9933-493c-8caa-c1a2dae789e4-metrics-client-ca\") pod \"telemeter-client-8b96c8b44-w7j7n\" (UID: \"f7d083fa-9933-493c-8caa-c1a2dae789e4\") " pod="openshift-monitoring/telemeter-client-8b96c8b44-w7j7n"
Sep 16 12:42:53.834895 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.834809    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7d083fa-9933-493c-8caa-c1a2dae789e4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8b96c8b44-w7j7n\" (UID: \"f7d083fa-9933-493c-8caa-c1a2dae789e4\") " pod="openshift-monitoring/telemeter-client-8b96c8b44-w7j7n"
Sep 16 12:42:53.834895 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.834846    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f7d083fa-9933-493c-8caa-c1a2dae789e4-federate-client-tls\") pod \"telemeter-client-8b96c8b44-w7j7n\" (UID: \"f7d083fa-9933-493c-8caa-c1a2dae789e4\") " pod="openshift-monitoring/telemeter-client-8b96c8b44-w7j7n"
Sep 16 12:42:53.834895 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.834866    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f7d083fa-9933-493c-8caa-c1a2dae789e4-telemeter-client-tls\") pod \"telemeter-client-8b96c8b44-w7j7n\" (UID: \"f7d083fa-9933-493c-8caa-c1a2dae789e4\") " pod="openshift-monitoring/telemeter-client-8b96c8b44-w7j7n"
Sep 16 12:42:53.835156 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.834980    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7d083fa-9933-493c-8caa-c1a2dae789e4-serving-certs-ca-bundle\") pod \"telemeter-client-8b96c8b44-w7j7n\" (UID: \"f7d083fa-9933-493c-8caa-c1a2dae789e4\") " pod="openshift-monitoring/telemeter-client-8b96c8b44-w7j7n"
Sep 16 12:42:53.835156 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.835059    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7d083fa-9933-493c-8caa-c1a2dae789e4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8b96c8b44-w7j7n\" (UID: \"f7d083fa-9933-493c-8caa-c1a2dae789e4\") " pod="openshift-monitoring/telemeter-client-8b96c8b44-w7j7n"
Sep 16 12:42:53.835156 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.835096    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgbv4\" (UniqueName: \"kubernetes.io/projected/f7d083fa-9933-493c-8caa-c1a2dae789e4-kube-api-access-tgbv4\") pod \"telemeter-client-8b96c8b44-w7j7n\" (UID: \"f7d083fa-9933-493c-8caa-c1a2dae789e4\") " pod="openshift-monitoring/telemeter-client-8b96c8b44-w7j7n"
Sep 16 12:42:53.835156 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.835135    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7d083fa-9933-493c-8caa-c1a2dae789e4-metrics-client-ca\") pod \"telemeter-client-8b96c8b44-w7j7n\" (UID: \"f7d083fa-9933-493c-8caa-c1a2dae789e4\") " pod="openshift-monitoring/telemeter-client-8b96c8b44-w7j7n"
Sep 16 12:42:53.835356 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.835316    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f7d083fa-9933-493c-8caa-c1a2dae789e4-secret-telemeter-client\") pod \"telemeter-client-8b96c8b44-w7j7n\" (UID: \"f7d083fa-9933-493c-8caa-c1a2dae789e4\") " pod="openshift-monitoring/telemeter-client-8b96c8b44-w7j7n"
Sep 16 12:42:53.835741 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.835705    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7d083fa-9933-493c-8caa-c1a2dae789e4-serving-certs-ca-bundle\") pod \"telemeter-client-8b96c8b44-w7j7n\" (UID: \"f7d083fa-9933-493c-8caa-c1a2dae789e4\") " pod="openshift-monitoring/telemeter-client-8b96c8b44-w7j7n"
Sep 16 12:42:53.836000 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.835979    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7d083fa-9933-493c-8caa-c1a2dae789e4-metrics-client-ca\") pod \"telemeter-client-8b96c8b44-w7j7n\" (UID: \"f7d083fa-9933-493c-8caa-c1a2dae789e4\") " pod="openshift-monitoring/telemeter-client-8b96c8b44-w7j7n"
Sep 16 12:42:53.836233 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.836213    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7d083fa-9933-493c-8caa-c1a2dae789e4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8b96c8b44-w7j7n\" (UID: \"f7d083fa-9933-493c-8caa-c1a2dae789e4\") " pod="openshift-monitoring/telemeter-client-8b96c8b44-w7j7n"
Sep 16 12:42:53.837706 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.837683    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7d083fa-9933-493c-8caa-c1a2dae789e4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8b96c8b44-w7j7n\" (UID: \"f7d083fa-9933-493c-8caa-c1a2dae789e4\") " pod="openshift-monitoring/telemeter-client-8b96c8b44-w7j7n"
Sep 16 12:42:53.837793 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.837774    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f7d083fa-9933-493c-8caa-c1a2dae789e4-secret-telemeter-client\") pod \"telemeter-client-8b96c8b44-w7j7n\" (UID: \"f7d083fa-9933-493c-8caa-c1a2dae789e4\") " pod="openshift-monitoring/telemeter-client-8b96c8b44-w7j7n"
Sep 16 12:42:53.837967 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.837946    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f7d083fa-9933-493c-8caa-c1a2dae789e4-telemeter-client-tls\") pod \"telemeter-client-8b96c8b44-w7j7n\" (UID: \"f7d083fa-9933-493c-8caa-c1a2dae789e4\") " pod="openshift-monitoring/telemeter-client-8b96c8b44-w7j7n"
Sep 16 12:42:53.838121 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.838100    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f7d083fa-9933-493c-8caa-c1a2dae789e4-federate-client-tls\") pod \"telemeter-client-8b96c8b44-w7j7n\" (UID: \"f7d083fa-9933-493c-8caa-c1a2dae789e4\") " pod="openshift-monitoring/telemeter-client-8b96c8b44-w7j7n"
Sep 16 12:42:53.845692 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.845671    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgbv4\" (UniqueName: \"kubernetes.io/projected/f7d083fa-9933-493c-8caa-c1a2dae789e4-kube-api-access-tgbv4\") pod \"telemeter-client-8b96c8b44-w7j7n\" (UID: \"f7d083fa-9933-493c-8caa-c1a2dae789e4\") " pod="openshift-monitoring/telemeter-client-8b96c8b44-w7j7n"
Sep 16 12:42:53.892458 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:53.892430    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-8b96c8b44-w7j7n"
Sep 16 12:42:54.192922 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:54.192824    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-8b96c8b44-w7j7n"]
Sep 16 12:42:54.199140 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:42:54.199112    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d083fa_9933_493c_8caa_c1a2dae789e4.slice/crio-7813a36f332305caae7688aa2f0dcb2f8e66808113001832766ef824260aea0e WatchSource:0}: Error finding container 7813a36f332305caae7688aa2f0dcb2f8e66808113001832766ef824260aea0e: Status 404 returned error can't find the container with id 7813a36f332305caae7688aa2f0dcb2f8e66808113001832766ef824260aea0e
Sep 16 12:42:55.105734 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:55.105700    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8b96c8b44-w7j7n" event={"ID":"f7d083fa-9933-493c-8caa-c1a2dae789e4","Type":"ContainerStarted","Data":"7813a36f332305caae7688aa2f0dcb2f8e66808113001832766ef824260aea0e"}
Sep 16 12:42:56.110725 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:56.110688    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8b96c8b44-w7j7n" event={"ID":"f7d083fa-9933-493c-8caa-c1a2dae789e4","Type":"ContainerStarted","Data":"f158382d562abd2a22b3fdabfa2e855006e901020a485f42d5644a01a2d39c3d"}
Sep 16 12:42:58.121015 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:58.120977    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8b96c8b44-w7j7n" event={"ID":"f7d083fa-9933-493c-8caa-c1a2dae789e4","Type":"ContainerStarted","Data":"e46b3cae8c91099d4e408e12b3ab6886a42733118ac021ee79e89b53f15f3092"}
Sep 16 12:42:58.121747 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:58.121064    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8b96c8b44-w7j7n" event={"ID":"f7d083fa-9933-493c-8caa-c1a2dae789e4","Type":"ContainerStarted","Data":"c83f20cb5afe343c18a73d036722de43a230cee22a168d23752a54af1542db44"}
Sep 16 12:42:58.150358 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:42:58.150314    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-8b96c8b44-w7j7n" podStartSLOduration=2.216486176 podStartE2EDuration="5.150297264s" totalImagesPullingTime="2.933811088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:42:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:42:58.149065463 +0000 UTC m=+284.596298107" watchObservedRunningTime="2026-09-16 12:42:58.150297264 +0000 UTC m=+284.597529914"
Sep 16 12:43:13.994464 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:43:13.994435    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vbt8d_0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174/console-operator/1.log"
Sep 16 12:43:13.995776 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:43:13.995755    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vbt8d_0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174/console-operator/1.log"
Sep 16 12:43:14.015575 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:43:14.015552    2577 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 12:44:03.829791 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:03.829757    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-689cccd495-wnwvg"]
Sep 16 12:44:03.833221 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:03.833191    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-689cccd495-wnwvg"
Sep 16 12:44:03.836250 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:03.836225    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 12:44:03.836538 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:03.836517    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 12:44:03.837433 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:03.837411    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 12:44:03.837547 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:03.837470    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 12:44:03.837930 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:03.837912    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 12:44:03.838050 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:03.837938    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jjnxd\""
Sep 16 12:44:03.842101 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:03.842083    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 12:44:03.846301 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:03.846282    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-689cccd495-wnwvg"]
Sep 16 12:44:03.977353 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:03.977317    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkd5h\" (UniqueName: \"kubernetes.io/projected/2a290979-8aa8-45cc-b7cd-4942036b2c5b-kube-api-access-xkd5h\") pod \"console-689cccd495-wnwvg\" (UID: \"2a290979-8aa8-45cc-b7cd-4942036b2c5b\") " pod="openshift-console/console-689cccd495-wnwvg"
Sep 16 12:44:03.977524 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:03.977365    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a290979-8aa8-45cc-b7cd-4942036b2c5b-trusted-ca-bundle\") pod \"console-689cccd495-wnwvg\" (UID: \"2a290979-8aa8-45cc-b7cd-4942036b2c5b\") " pod="openshift-console/console-689cccd495-wnwvg"
Sep 16 12:44:03.977524 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:03.977384    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a290979-8aa8-45cc-b7cd-4942036b2c5b-service-ca\") pod \"console-689cccd495-wnwvg\" (UID: \"2a290979-8aa8-45cc-b7cd-4942036b2c5b\") " pod="openshift-console/console-689cccd495-wnwvg"
Sep 16 12:44:03.977524 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:03.977473    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a290979-8aa8-45cc-b7cd-4942036b2c5b-console-oauth-config\") pod \"console-689cccd495-wnwvg\" (UID: \"2a290979-8aa8-45cc-b7cd-4942036b2c5b\") " pod="openshift-console/console-689cccd495-wnwvg"
Sep 16 12:44:03.977524 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:03.977517    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a290979-8aa8-45cc-b7cd-4942036b2c5b-console-serving-cert\") pod \"console-689cccd495-wnwvg\" (UID: \"2a290979-8aa8-45cc-b7cd-4942036b2c5b\") " pod="openshift-console/console-689cccd495-wnwvg"
Sep 16 12:44:03.977652 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:03.977534    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a290979-8aa8-45cc-b7cd-4942036b2c5b-oauth-serving-cert\") pod \"console-689cccd495-wnwvg\" (UID: \"2a290979-8aa8-45cc-b7cd-4942036b2c5b\") " pod="openshift-console/console-689cccd495-wnwvg"
Sep 16 12:44:03.977652 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:03.977578    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a290979-8aa8-45cc-b7cd-4942036b2c5b-console-config\") pod \"console-689cccd495-wnwvg\" (UID: \"2a290979-8aa8-45cc-b7cd-4942036b2c5b\") " pod="openshift-console/console-689cccd495-wnwvg"
Sep 16 12:44:04.078919 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:04.078888    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkd5h\" (UniqueName: \"kubernetes.io/projected/2a290979-8aa8-45cc-b7cd-4942036b2c5b-kube-api-access-xkd5h\") pod \"console-689cccd495-wnwvg\" (UID: \"2a290979-8aa8-45cc-b7cd-4942036b2c5b\") " pod="openshift-console/console-689cccd495-wnwvg"
Sep 16 12:44:04.079104 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:04.078927    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a290979-8aa8-45cc-b7cd-4942036b2c5b-trusted-ca-bundle\") pod \"console-689cccd495-wnwvg\" (UID: \"2a290979-8aa8-45cc-b7cd-4942036b2c5b\") " pod="openshift-console/console-689cccd495-wnwvg"
Sep 16 12:44:04.079104 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:04.078946    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a290979-8aa8-45cc-b7cd-4942036b2c5b-service-ca\") pod \"console-689cccd495-wnwvg\" (UID: \"2a290979-8aa8-45cc-b7cd-4942036b2c5b\") " pod="openshift-console/console-689cccd495-wnwvg"
Sep 16 12:44:04.079104 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:04.078977    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a290979-8aa8-45cc-b7cd-4942036b2c5b-console-oauth-config\") pod \"console-689cccd495-wnwvg\" (UID: \"2a290979-8aa8-45cc-b7cd-4942036b2c5b\") " pod="openshift-console/console-689cccd495-wnwvg"
Sep 16 12:44:04.079104 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:04.078999    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a290979-8aa8-45cc-b7cd-4942036b2c5b-console-serving-cert\") pod \"console-689cccd495-wnwvg\" (UID: \"2a290979-8aa8-45cc-b7cd-4942036b2c5b\") " pod="openshift-console/console-689cccd495-wnwvg"
Sep 16 12:44:04.079104 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:04.079013    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a290979-8aa8-45cc-b7cd-4942036b2c5b-oauth-serving-cert\") pod \"console-689cccd495-wnwvg\" (UID: \"2a290979-8aa8-45cc-b7cd-4942036b2c5b\") " pod="openshift-console/console-689cccd495-wnwvg"
Sep 16 12:44:04.079104 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:04.079046    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a290979-8aa8-45cc-b7cd-4942036b2c5b-console-config\") pod \"console-689cccd495-wnwvg\" (UID: \"2a290979-8aa8-45cc-b7cd-4942036b2c5b\") " pod="openshift-console/console-689cccd495-wnwvg"
Sep 16 12:44:04.079763 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:04.079741    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a290979-8aa8-45cc-b7cd-4942036b2c5b-oauth-serving-cert\") pod \"console-689cccd495-wnwvg\" (UID: \"2a290979-8aa8-45cc-b7cd-4942036b2c5b\") " pod="openshift-console/console-689cccd495-wnwvg"
Sep 16 12:44:04.079887 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:04.079797    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a290979-8aa8-45cc-b7cd-4942036b2c5b-service-ca\") pod \"console-689cccd495-wnwvg\" (UID: \"2a290979-8aa8-45cc-b7cd-4942036b2c5b\") " pod="openshift-console/console-689cccd495-wnwvg"
Sep 16 12:44:04.079887 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:04.079828    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a290979-8aa8-45cc-b7cd-4942036b2c5b-console-config\") pod \"console-689cccd495-wnwvg\" (UID: \"2a290979-8aa8-45cc-b7cd-4942036b2c5b\") " pod="openshift-console/console-689cccd495-wnwvg"
Sep 16 12:44:04.080346 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:04.080324    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a290979-8aa8-45cc-b7cd-4942036b2c5b-trusted-ca-bundle\") pod \"console-689cccd495-wnwvg\" (UID: \"2a290979-8aa8-45cc-b7cd-4942036b2c5b\") " pod="openshift-console/console-689cccd495-wnwvg"
Sep 16 12:44:04.081659 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:04.081641    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a290979-8aa8-45cc-b7cd-4942036b2c5b-console-serving-cert\") pod \"console-689cccd495-wnwvg\" (UID: \"2a290979-8aa8-45cc-b7cd-4942036b2c5b\") " pod="openshift-console/console-689cccd495-wnwvg"
Sep 16 12:44:04.081893 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:04.081874    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a290979-8aa8-45cc-b7cd-4942036b2c5b-console-oauth-config\") pod \"console-689cccd495-wnwvg\" (UID: \"2a290979-8aa8-45cc-b7cd-4942036b2c5b\") " pod="openshift-console/console-689cccd495-wnwvg"
Sep 16 12:44:04.088256 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:04.088233    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkd5h\" (UniqueName: \"kubernetes.io/projected/2a290979-8aa8-45cc-b7cd-4942036b2c5b-kube-api-access-xkd5h\") pod \"console-689cccd495-wnwvg\" (UID: \"2a290979-8aa8-45cc-b7cd-4942036b2c5b\") " pod="openshift-console/console-689cccd495-wnwvg"
Sep 16 12:44:04.144080 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:04.144050    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-689cccd495-wnwvg"
Sep 16 12:44:04.416951 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:04.416925    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-689cccd495-wnwvg"]
Sep 16 12:44:04.419667 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:44:04.419638    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a290979_8aa8_45cc_b7cd_4942036b2c5b.slice/crio-cecbf66037ee59af5bd5662d8dc470429d7528dafb2bf21fdef263ef9f700e29 WatchSource:0}: Error finding container cecbf66037ee59af5bd5662d8dc470429d7528dafb2bf21fdef263ef9f700e29: Status 404 returned error can't find the container with id cecbf66037ee59af5bd5662d8dc470429d7528dafb2bf21fdef263ef9f700e29
Sep 16 12:44:05.383106 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:05.383005    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-689cccd495-wnwvg" event={"ID":"2a290979-8aa8-45cc-b7cd-4942036b2c5b","Type":"ContainerStarted","Data":"8b338e7288a18980424d17c2e100e37a2f1bc6edf2ed8ef8d7249a486b4442fb"}
Sep 16 12:44:05.383106 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:05.383083    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-689cccd495-wnwvg" event={"ID":"2a290979-8aa8-45cc-b7cd-4942036b2c5b","Type":"ContainerStarted","Data":"cecbf66037ee59af5bd5662d8dc470429d7528dafb2bf21fdef263ef9f700e29"}
Sep 16 12:44:05.403271 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:05.403221    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-689cccd495-wnwvg" podStartSLOduration=2.403204377 podStartE2EDuration="2.403204377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:44:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:44:05.401576929 +0000 UTC m=+351.848809582" watchObservedRunningTime="2026-09-16 12:44:05.403204377 +0000 UTC m=+351.850437018"
Sep 16 12:44:14.144892 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:14.144860    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-689cccd495-wnwvg"
Sep 16 12:44:14.145290 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:14.144915    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-689cccd495-wnwvg"
Sep 16 12:44:14.149662 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:14.149639    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-689cccd495-wnwvg"
Sep 16 12:44:14.423454 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:44:14.423367    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-689cccd495-wnwvg"
Sep 16 12:45:54.192796 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:45:54.192765    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-285mw"]
Sep 16 12:45:54.195969 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:45:54.195951    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-285mw"
Sep 16 12:45:54.199128 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:45:54.199109    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 16 12:45:54.199128 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:45:54.199114    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:45:54.199298 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:45:54.199145    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-69957\""
Sep 16 12:45:54.208734 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:45:54.208713    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-285mw"]
Sep 16 12:45:54.233530 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:45:54.233465    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42edbbc7-6435-4036-be70-f3fdc455dd79-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-285mw\" (UID: \"42edbbc7-6435-4036-be70-f3fdc455dd79\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-285mw"
Sep 16 12:45:54.233646 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:45:54.233631    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7vfc\" (UniqueName: \"kubernetes.io/projected/42edbbc7-6435-4036-be70-f3fdc455dd79-kube-api-access-r7vfc\") pod \"cert-manager-operator-controller-manager-55c444bf86-285mw\" (UID: \"42edbbc7-6435-4036-be70-f3fdc455dd79\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-285mw"
Sep 16 12:45:54.334513 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:45:54.334477    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42edbbc7-6435-4036-be70-f3fdc455dd79-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-285mw\" (UID: \"42edbbc7-6435-4036-be70-f3fdc455dd79\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-285mw"
Sep 16 12:45:54.334665 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:45:54.334564    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7vfc\" (UniqueName: \"kubernetes.io/projected/42edbbc7-6435-4036-be70-f3fdc455dd79-kube-api-access-r7vfc\") pod \"cert-manager-operator-controller-manager-55c444bf86-285mw\" (UID: \"42edbbc7-6435-4036-be70-f3fdc455dd79\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-285mw"
Sep 16 12:45:54.334953 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:45:54.334935    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42edbbc7-6435-4036-be70-f3fdc455dd79-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-285mw\" (UID: \"42edbbc7-6435-4036-be70-f3fdc455dd79\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-285mw"
Sep 16 12:45:54.342799 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:45:54.342770    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7vfc\" (UniqueName: \"kubernetes.io/projected/42edbbc7-6435-4036-be70-f3fdc455dd79-kube-api-access-r7vfc\") pod \"cert-manager-operator-controller-manager-55c444bf86-285mw\" (UID: \"42edbbc7-6435-4036-be70-f3fdc455dd79\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-285mw"
Sep 16 12:45:54.505077 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:45:54.505053    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-285mw"
Sep 16 12:45:54.780555 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:45:54.780528    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-285mw"]
Sep 16 12:45:54.782619 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:45:54.782593    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42edbbc7_6435_4036_be70_f3fdc455dd79.slice/crio-f994ddef478bbad6c1689333d7f643441f1e6537a5deb034789456d77c803cac WatchSource:0}: Error finding container f994ddef478bbad6c1689333d7f643441f1e6537a5deb034789456d77c803cac: Status 404 returned error can't find the container with id f994ddef478bbad6c1689333d7f643441f1e6537a5deb034789456d77c803cac
Sep 16 12:45:54.786000 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:45:54.785982    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:45:54.808245 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:45:54.808217    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-285mw" event={"ID":"42edbbc7-6435-4036-be70-f3fdc455dd79","Type":"ContainerStarted","Data":"f994ddef478bbad6c1689333d7f643441f1e6537a5deb034789456d77c803cac"}
Sep 16 12:45:57.823705 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:45:57.823667    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-285mw" event={"ID":"42edbbc7-6435-4036-be70-f3fdc455dd79","Type":"ContainerStarted","Data":"75ad672c038711abf0c6ef70b6ef32b931b03c14135472e87342d8ac38598a73"}
Sep 16 12:45:57.851379 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:45:57.851335    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-285mw" podStartSLOduration=1.3348048399999999 podStartE2EDuration="3.851319273s" totalImagesPullingTime="2.516514433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:45:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:45:57.848734703 +0000 UTC m=+464.295967383" watchObservedRunningTime="2026-09-16 12:45:57.851319273 +0000 UTC m=+464.298551914"
Sep 16 12:45:58.642934 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:45:58.642902    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6d9vf"]
Sep 16 12:45:58.646425 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:45:58.646403    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-6d9vf"
Sep 16 12:45:58.649699 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:45:58.649674    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 16 12:45:58.649699 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:45:58.649675    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-cllfh\""
Sep 16 12:45:58.651016 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:45:58.650998    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 16 12:45:58.662249 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:45:58.662230    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6d9vf"]
Sep 16 12:45:58.772952 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:45:58.772919    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6863a82f-5694-47e7-9e9e-fab8847eaac2-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6d9vf\" (UID: \"6863a82f-5694-47e7-9e9e-fab8847eaac2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6d9vf"
Sep 16 12:45:58.773134 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:45:58.772999    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m62d2\" (UniqueName: \"kubernetes.io/projected/6863a82f-5694-47e7-9e9e-fab8847eaac2-kube-api-access-m62d2\") pod \"cert-manager-webhook-6486ffd997-6d9vf\" (UID: \"6863a82f-5694-47e7-9e9e-fab8847eaac2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6d9vf"
Sep 16 12:45:58.874423 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:45:58.874390    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m62d2\" (UniqueName: \"kubernetes.io/projected/6863a82f-5694-47e7-9e9e-fab8847eaac2-kube-api-access-m62d2\") pod \"cert-manager-webhook-6486ffd997-6d9vf\" (UID: \"6863a82f-5694-47e7-9e9e-fab8847eaac2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6d9vf"
Sep 16 12:45:58.874882 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:45:58.874442    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6863a82f-5694-47e7-9e9e-fab8847eaac2-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6d9vf\" (UID: \"6863a82f-5694-47e7-9e9e-fab8847eaac2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6d9vf"
Sep 16 12:45:58.884489 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:45:58.884465    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m62d2\" (UniqueName: \"kubernetes.io/projected/6863a82f-5694-47e7-9e9e-fab8847eaac2-kube-api-access-m62d2\") pod \"cert-manager-webhook-6486ffd997-6d9vf\" (UID: \"6863a82f-5694-47e7-9e9e-fab8847eaac2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6d9vf"
Sep 16 12:45:58.884596 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:45:58.884507    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6863a82f-5694-47e7-9e9e-fab8847eaac2-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6d9vf\" (UID: \"6863a82f-5694-47e7-9e9e-fab8847eaac2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6d9vf"
Sep 16 12:45:58.968720 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:45:58.968635    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-6d9vf"
Sep 16 12:45:59.240653 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:45:59.240580    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6d9vf"]
Sep 16 12:45:59.243808 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:45:59.243782    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6863a82f_5694_47e7_9e9e_fab8847eaac2.slice/crio-2481033c12895ea47af1ed0e10126789dc96c0b1fe6f91a2e4b8e8f6ea265197 WatchSource:0}: Error finding container 2481033c12895ea47af1ed0e10126789dc96c0b1fe6f91a2e4b8e8f6ea265197: Status 404 returned error can't find the container with id 2481033c12895ea47af1ed0e10126789dc96c0b1fe6f91a2e4b8e8f6ea265197
Sep 16 12:45:59.834088 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:45:59.834056    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-6d9vf" event={"ID":"6863a82f-5694-47e7-9e9e-fab8847eaac2","Type":"ContainerStarted","Data":"2481033c12895ea47af1ed0e10126789dc96c0b1fe6f91a2e4b8e8f6ea265197"}
Sep 16 12:46:02.850132 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:02.850102    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-6d9vf" event={"ID":"6863a82f-5694-47e7-9e9e-fab8847eaac2","Type":"ContainerStarted","Data":"6f0bf41dfff5c103addeb526ee22ff9a3fcd16e98c27de05539a29a1a3af0ec5"}
Sep 16 12:46:02.850545 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:02.850229    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-6d9vf"
Sep 16 12:46:02.868872 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:02.868824    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-6d9vf" podStartSLOduration=1.7556676530000002 podStartE2EDuration="4.868809901s" totalImagesPullingTime="3.113142248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:45:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:46:02.868149951 +0000 UTC m=+469.315382602" watchObservedRunningTime="2026-09-16 12:46:02.868809901 +0000 UTC m=+469.316042542"
Sep 16 12:46:06.353682 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:06.353645    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zpzmb"]
Sep 16 12:46:06.357355 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:06.357338    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zpzmb"
Sep 16 12:46:06.360671 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:06.360648    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-6lj7q\""
Sep 16 12:46:06.368012 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:06.367989    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zpzmb"]
Sep 16 12:46:06.446133 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:06.446101    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/90871eae-0117-43cb-b28a-79fb0c21a260-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zpzmb\" (UID: \"90871eae-0117-43cb-b28a-79fb0c21a260\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zpzmb"
Sep 16 12:46:06.446286 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:06.446179    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dxx2\" (UniqueName: \"kubernetes.io/projected/90871eae-0117-43cb-b28a-79fb0c21a260-kube-api-access-7dxx2\") pod \"cert-manager-cainjector-559cdd8c87-zpzmb\" (UID: \"90871eae-0117-43cb-b28a-79fb0c21a260\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zpzmb"
Sep 16 12:46:06.546619 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:06.546581    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/90871eae-0117-43cb-b28a-79fb0c21a260-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zpzmb\" (UID: \"90871eae-0117-43cb-b28a-79fb0c21a260\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zpzmb"
Sep 16 12:46:06.546820 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:06.546669    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dxx2\" (UniqueName: \"kubernetes.io/projected/90871eae-0117-43cb-b28a-79fb0c21a260-kube-api-access-7dxx2\") pod \"cert-manager-cainjector-559cdd8c87-zpzmb\" (UID: \"90871eae-0117-43cb-b28a-79fb0c21a260\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zpzmb"
Sep 16 12:46:06.556315 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:06.556286    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dxx2\" (UniqueName: \"kubernetes.io/projected/90871eae-0117-43cb-b28a-79fb0c21a260-kube-api-access-7dxx2\") pod \"cert-manager-cainjector-559cdd8c87-zpzmb\" (UID: \"90871eae-0117-43cb-b28a-79fb0c21a260\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zpzmb"
Sep 16 12:46:06.556424 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:06.556286    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/90871eae-0117-43cb-b28a-79fb0c21a260-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zpzmb\" (UID: \"90871eae-0117-43cb-b28a-79fb0c21a260\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zpzmb"
Sep 16 12:46:06.666015 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:06.665927    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zpzmb"
Sep 16 12:46:06.939464 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:06.939441    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zpzmb"]
Sep 16 12:46:06.941948 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:46:06.941921    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90871eae_0117_43cb_b28a_79fb0c21a260.slice/crio-595173547f3ce156f58cac26a89bd09f106dbc5d42cbd59ae336256ef49eb18b WatchSource:0}: Error finding container 595173547f3ce156f58cac26a89bd09f106dbc5d42cbd59ae336256ef49eb18b: Status 404 returned error can't find the container with id 595173547f3ce156f58cac26a89bd09f106dbc5d42cbd59ae336256ef49eb18b
Sep 16 12:46:07.872020 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:07.871986    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zpzmb" event={"ID":"90871eae-0117-43cb-b28a-79fb0c21a260","Type":"ContainerStarted","Data":"6c7b9111574b427ea500d34a3d0bbc0008914f22e531ff9402ca84c87624f71b"}
Sep 16 12:46:07.872020 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:07.872022    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zpzmb" event={"ID":"90871eae-0117-43cb-b28a-79fb0c21a260","Type":"ContainerStarted","Data":"595173547f3ce156f58cac26a89bd09f106dbc5d42cbd59ae336256ef49eb18b"}
Sep 16 12:46:07.892773 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:07.892723    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zpzmb" podStartSLOduration=1.892707727 podStartE2EDuration="1.892707727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:46:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:46:07.890580234 +0000 UTC m=+474.337812889" watchObservedRunningTime="2026-09-16 12:46:07.892707727 +0000 UTC m=+474.339940369"
Sep 16 12:46:08.856584 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:08.856546    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-6d9vf"
Sep 16 12:46:15.066768 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:15.066733    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55c45d8cc9-48v4z"]
Sep 16 12:46:15.070166 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:15.070142    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55c45d8cc9-48v4z"
Sep 16 12:46:15.085326 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:15.085301    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55c45d8cc9-48v4z"]
Sep 16 12:46:15.219586 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:15.219556    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca507687-8e56-4d00-8bd1-9aa1b80d489b-service-ca\") pod \"console-55c45d8cc9-48v4z\" (UID: \"ca507687-8e56-4d00-8bd1-9aa1b80d489b\") " pod="openshift-console/console-55c45d8cc9-48v4z"
Sep 16 12:46:15.219586 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:15.219587    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca507687-8e56-4d00-8bd1-9aa1b80d489b-trusted-ca-bundle\") pod \"console-55c45d8cc9-48v4z\" (UID: \"ca507687-8e56-4d00-8bd1-9aa1b80d489b\") " pod="openshift-console/console-55c45d8cc9-48v4z"
Sep 16 12:46:15.219803 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:15.219615    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca507687-8e56-4d00-8bd1-9aa1b80d489b-console-oauth-config\") pod \"console-55c45d8cc9-48v4z\" (UID: \"ca507687-8e56-4d00-8bd1-9aa1b80d489b\") " pod="openshift-console/console-55c45d8cc9-48v4z"
Sep 16 12:46:15.219803 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:15.219728    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca507687-8e56-4d00-8bd1-9aa1b80d489b-console-config\") pod \"console-55c45d8cc9-48v4z\" (UID: \"ca507687-8e56-4d00-8bd1-9aa1b80d489b\") " pod="openshift-console/console-55c45d8cc9-48v4z"
Sep 16 12:46:15.219803 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:15.219776    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp2qm\" (UniqueName: \"kubernetes.io/projected/ca507687-8e56-4d00-8bd1-9aa1b80d489b-kube-api-access-tp2qm\") pod \"console-55c45d8cc9-48v4z\" (UID: \"ca507687-8e56-4d00-8bd1-9aa1b80d489b\") " pod="openshift-console/console-55c45d8cc9-48v4z"
Sep 16 12:46:15.219958 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:15.219842    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca507687-8e56-4d00-8bd1-9aa1b80d489b-oauth-serving-cert\") pod \"console-55c45d8cc9-48v4z\" (UID: \"ca507687-8e56-4d00-8bd1-9aa1b80d489b\") " pod="openshift-console/console-55c45d8cc9-48v4z"
Sep 16 12:46:15.219958 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:15.219886    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca507687-8e56-4d00-8bd1-9aa1b80d489b-console-serving-cert\") pod \"console-55c45d8cc9-48v4z\" (UID: \"ca507687-8e56-4d00-8bd1-9aa1b80d489b\") " pod="openshift-console/console-55c45d8cc9-48v4z"
Sep 16 12:46:15.321096 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:15.320993    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca507687-8e56-4d00-8bd1-9aa1b80d489b-console-serving-cert\") pod \"console-55c45d8cc9-48v4z\" (UID: \"ca507687-8e56-4d00-8bd1-9aa1b80d489b\") " pod="openshift-console/console-55c45d8cc9-48v4z"
Sep 16 12:46:15.321253 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:15.321099    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca507687-8e56-4d00-8bd1-9aa1b80d489b-service-ca\") pod \"console-55c45d8cc9-48v4z\" (UID: \"ca507687-8e56-4d00-8bd1-9aa1b80d489b\") " pod="openshift-console/console-55c45d8cc9-48v4z"
Sep 16 12:46:15.321253 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:15.321128    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca507687-8e56-4d00-8bd1-9aa1b80d489b-trusted-ca-bundle\") pod \"console-55c45d8cc9-48v4z\" (UID: \"ca507687-8e56-4d00-8bd1-9aa1b80d489b\") " pod="openshift-console/console-55c45d8cc9-48v4z"
Sep 16 12:46:15.321253 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:15.321164    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca507687-8e56-4d00-8bd1-9aa1b80d489b-console-oauth-config\") pod \"console-55c45d8cc9-48v4z\" (UID: \"ca507687-8e56-4d00-8bd1-9aa1b80d489b\") " pod="openshift-console/console-55c45d8cc9-48v4z"
Sep 16 12:46:15.321253 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:15.321209    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca507687-8e56-4d00-8bd1-9aa1b80d489b-console-config\") pod \"console-55c45d8cc9-48v4z\" (UID: \"ca507687-8e56-4d00-8bd1-9aa1b80d489b\") " pod="openshift-console/console-55c45d8cc9-48v4z"
Sep 16 12:46:15.321253 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:15.321245    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp2qm\" (UniqueName: \"kubernetes.io/projected/ca507687-8e56-4d00-8bd1-9aa1b80d489b-kube-api-access-tp2qm\") pod \"console-55c45d8cc9-48v4z\" (UID: \"ca507687-8e56-4d00-8bd1-9aa1b80d489b\") " pod="openshift-console/console-55c45d8cc9-48v4z"
Sep 16 12:46:15.321488 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:15.321280    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca507687-8e56-4d00-8bd1-9aa1b80d489b-oauth-serving-cert\") pod \"console-55c45d8cc9-48v4z\" (UID: \"ca507687-8e56-4d00-8bd1-9aa1b80d489b\") " pod="openshift-console/console-55c45d8cc9-48v4z"
Sep 16 12:46:15.322007 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:15.321980    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca507687-8e56-4d00-8bd1-9aa1b80d489b-console-config\") pod \"console-55c45d8cc9-48v4z\" (UID: \"ca507687-8e56-4d00-8bd1-9aa1b80d489b\") " pod="openshift-console/console-55c45d8cc9-48v4z"
Sep 16 12:46:15.322130 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:15.321990    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca507687-8e56-4d00-8bd1-9aa1b80d489b-service-ca\") pod \"console-55c45d8cc9-48v4z\" (UID: \"ca507687-8e56-4d00-8bd1-9aa1b80d489b\") " pod="openshift-console/console-55c45d8cc9-48v4z"
Sep 16 12:46:15.322130 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:15.322074    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca507687-8e56-4d00-8bd1-9aa1b80d489b-trusted-ca-bundle\") pod \"console-55c45d8cc9-48v4z\" (UID: \"ca507687-8e56-4d00-8bd1-9aa1b80d489b\") " pod="openshift-console/console-55c45d8cc9-48v4z"
Sep 16 12:46:15.322241 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:15.322219    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca507687-8e56-4d00-8bd1-9aa1b80d489b-oauth-serving-cert\") pod \"console-55c45d8cc9-48v4z\" (UID: \"ca507687-8e56-4d00-8bd1-9aa1b80d489b\") " pod="openshift-console/console-55c45d8cc9-48v4z"
Sep 16 12:46:15.323712 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:15.323690    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca507687-8e56-4d00-8bd1-9aa1b80d489b-console-oauth-config\") pod \"console-55c45d8cc9-48v4z\" (UID: \"ca507687-8e56-4d00-8bd1-9aa1b80d489b\") " pod="openshift-console/console-55c45d8cc9-48v4z"
Sep 16 12:46:15.323827 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:15.323810    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca507687-8e56-4d00-8bd1-9aa1b80d489b-console-serving-cert\") pod \"console-55c45d8cc9-48v4z\" (UID: \"ca507687-8e56-4d00-8bd1-9aa1b80d489b\") " pod="openshift-console/console-55c45d8cc9-48v4z"
Sep 16 12:46:15.329482 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:15.329458    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp2qm\" (UniqueName: \"kubernetes.io/projected/ca507687-8e56-4d00-8bd1-9aa1b80d489b-kube-api-access-tp2qm\") pod \"console-55c45d8cc9-48v4z\" (UID: \"ca507687-8e56-4d00-8bd1-9aa1b80d489b\") " pod="openshift-console/console-55c45d8cc9-48v4z"
Sep 16 12:46:15.380494 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:15.380467    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55c45d8cc9-48v4z"
Sep 16 12:46:15.653349 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:15.653231    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55c45d8cc9-48v4z"]
Sep 16 12:46:15.655747 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:46:15.655719    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca507687_8e56_4d00_8bd1_9aa1b80d489b.slice/crio-c6affcdb169a1a54d3d0266a9f27b1518ae19bc47eddadc4f1e2602ca70174d5 WatchSource:0}: Error finding container c6affcdb169a1a54d3d0266a9f27b1518ae19bc47eddadc4f1e2602ca70174d5: Status 404 returned error can't find the container with id c6affcdb169a1a54d3d0266a9f27b1518ae19bc47eddadc4f1e2602ca70174d5
Sep 16 12:46:15.908107 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:15.908013    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55c45d8cc9-48v4z" event={"ID":"ca507687-8e56-4d00-8bd1-9aa1b80d489b","Type":"ContainerStarted","Data":"94ce0636881e9c8e0ecf4647fd545d9a421d93eb9e0378ee63da4cf69efebaab"}
Sep 16 12:46:15.908107 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:15.908090    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55c45d8cc9-48v4z" event={"ID":"ca507687-8e56-4d00-8bd1-9aa1b80d489b","Type":"ContainerStarted","Data":"c6affcdb169a1a54d3d0266a9f27b1518ae19bc47eddadc4f1e2602ca70174d5"}
Sep 16 12:46:15.932962 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:15.930799    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55c45d8cc9-48v4z" podStartSLOduration=0.930778805 podStartE2EDuration="930.778805ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:46:15.92846199 +0000 UTC m=+482.375694636" watchObservedRunningTime="2026-09-16 12:46:15.930778805 +0000 UTC m=+482.378011448"
Sep 16 12:46:25.380822 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:25.380788    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55c45d8cc9-48v4z"
Sep 16 12:46:25.381322 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:25.380850    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55c45d8cc9-48v4z"
Sep 16 12:46:25.385462 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:25.385439    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55c45d8cc9-48v4z"
Sep 16 12:46:25.956938 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:25.956912    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55c45d8cc9-48v4z"
Sep 16 12:46:26.013136 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:26.013103    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-689cccd495-wnwvg"]
Sep 16 12:46:37.594783 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:37.594747    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-575fbb86f4-qr6g8"]
Sep 16 12:46:37.597903 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:37.597887    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-575fbb86f4-qr6g8"
Sep 16 12:46:37.602410 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:37.602313    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:46:37.602410 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:37.602336    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 16 12:46:37.602410 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:37.602324    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 12:46:37.602410 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:37.602375    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 16 12:46:37.602410 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:37.602411    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-xqc5b\""
Sep 16 12:46:37.602710 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:37.602373    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 16 12:46:37.615380 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:37.615359    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-575fbb86f4-qr6g8"]
Sep 16 12:46:37.711759 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:37.711725    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1c9eb370-b426-4f43-ad86-7be180866403-metrics-cert\") pod \"lws-controller-manager-575fbb86f4-qr6g8\" (UID: \"1c9eb370-b426-4f43-ad86-7be180866403\") " pod="openshift-lws-operator/lws-controller-manager-575fbb86f4-qr6g8"
Sep 16 12:46:37.711759 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:37.711760    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnmrt\" (UniqueName: \"kubernetes.io/projected/1c9eb370-b426-4f43-ad86-7be180866403-kube-api-access-gnmrt\") pod \"lws-controller-manager-575fbb86f4-qr6g8\" (UID: \"1c9eb370-b426-4f43-ad86-7be180866403\") " pod="openshift-lws-operator/lws-controller-manager-575fbb86f4-qr6g8"
Sep 16 12:46:37.711978 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:37.711828    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1c9eb370-b426-4f43-ad86-7be180866403-manager-config\") pod \"lws-controller-manager-575fbb86f4-qr6g8\" (UID: \"1c9eb370-b426-4f43-ad86-7be180866403\") " pod="openshift-lws-operator/lws-controller-manager-575fbb86f4-qr6g8"
Sep 16 12:46:37.711978 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:37.711858    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c9eb370-b426-4f43-ad86-7be180866403-cert\") pod \"lws-controller-manager-575fbb86f4-qr6g8\" (UID: \"1c9eb370-b426-4f43-ad86-7be180866403\") " pod="openshift-lws-operator/lws-controller-manager-575fbb86f4-qr6g8"
Sep 16 12:46:37.813236 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:37.813199    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1c9eb370-b426-4f43-ad86-7be180866403-manager-config\") pod \"lws-controller-manager-575fbb86f4-qr6g8\" (UID: \"1c9eb370-b426-4f43-ad86-7be180866403\") " pod="openshift-lws-operator/lws-controller-manager-575fbb86f4-qr6g8"
Sep 16 12:46:37.813236 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:37.813233    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c9eb370-b426-4f43-ad86-7be180866403-cert\") pod \"lws-controller-manager-575fbb86f4-qr6g8\" (UID: \"1c9eb370-b426-4f43-ad86-7be180866403\") " pod="openshift-lws-operator/lws-controller-manager-575fbb86f4-qr6g8"
Sep 16 12:46:37.813500 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:37.813294    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1c9eb370-b426-4f43-ad86-7be180866403-metrics-cert\") pod \"lws-controller-manager-575fbb86f4-qr6g8\" (UID: \"1c9eb370-b426-4f43-ad86-7be180866403\") " pod="openshift-lws-operator/lws-controller-manager-575fbb86f4-qr6g8"
Sep 16 12:46:37.813500 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:37.813311    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnmrt\" (UniqueName: \"kubernetes.io/projected/1c9eb370-b426-4f43-ad86-7be180866403-kube-api-access-gnmrt\") pod \"lws-controller-manager-575fbb86f4-qr6g8\" (UID: \"1c9eb370-b426-4f43-ad86-7be180866403\") " pod="openshift-lws-operator/lws-controller-manager-575fbb86f4-qr6g8"
Sep 16 12:46:37.814000 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:37.813977    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1c9eb370-b426-4f43-ad86-7be180866403-manager-config\") pod \"lws-controller-manager-575fbb86f4-qr6g8\" (UID: \"1c9eb370-b426-4f43-ad86-7be180866403\") " pod="openshift-lws-operator/lws-controller-manager-575fbb86f4-qr6g8"
Sep 16 12:46:37.816008 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:37.815986    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1c9eb370-b426-4f43-ad86-7be180866403-metrics-cert\") pod \"lws-controller-manager-575fbb86f4-qr6g8\" (UID: \"1c9eb370-b426-4f43-ad86-7be180866403\") " pod="openshift-lws-operator/lws-controller-manager-575fbb86f4-qr6g8"
Sep 16 12:46:37.816126 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:37.816080    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c9eb370-b426-4f43-ad86-7be180866403-cert\") pod \"lws-controller-manager-575fbb86f4-qr6g8\" (UID: \"1c9eb370-b426-4f43-ad86-7be180866403\") " pod="openshift-lws-operator/lws-controller-manager-575fbb86f4-qr6g8"
Sep 16 12:46:37.822844 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:37.822824    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnmrt\" (UniqueName: \"kubernetes.io/projected/1c9eb370-b426-4f43-ad86-7be180866403-kube-api-access-gnmrt\") pod \"lws-controller-manager-575fbb86f4-qr6g8\" (UID: \"1c9eb370-b426-4f43-ad86-7be180866403\") " pod="openshift-lws-operator/lws-controller-manager-575fbb86f4-qr6g8"
Sep 16 12:46:37.908438 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:37.908366    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-575fbb86f4-qr6g8"
Sep 16 12:46:38.191810 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:38.191674    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-575fbb86f4-qr6g8"]
Sep 16 12:46:38.194684 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:46:38.194658    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c9eb370_b426_4f43_ad86_7be180866403.slice/crio-512b1f625c57038b8bf750a1138d02af45668fbc89a81e9d62242e9addf0ae82 WatchSource:0}: Error finding container 512b1f625c57038b8bf750a1138d02af45668fbc89a81e9d62242e9addf0ae82: Status 404 returned error can't find the container with id 512b1f625c57038b8bf750a1138d02af45668fbc89a81e9d62242e9addf0ae82
Sep 16 12:46:39.012423 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:39.012391    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-575fbb86f4-qr6g8" event={"ID":"1c9eb370-b426-4f43-ad86-7be180866403","Type":"ContainerStarted","Data":"512b1f625c57038b8bf750a1138d02af45668fbc89a81e9d62242e9addf0ae82"}
Sep 16 12:46:43.030405 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:43.030372    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-575fbb86f4-qr6g8" event={"ID":"1c9eb370-b426-4f43-ad86-7be180866403","Type":"ContainerStarted","Data":"b4d55c4227e124d6c798ebed50f54e52cc91465fc4ca2b263d71b1f318eeee1e"}
Sep 16 12:46:43.030771 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:43.030449    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-575fbb86f4-qr6g8"
Sep 16 12:46:43.060259 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:43.060210    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-575fbb86f4-qr6g8" podStartSLOduration=2.262305521 podStartE2EDuration="6.060195034s" totalImagesPullingTime="3.797889513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:46:43.058656605 +0000 UTC m=+509.505889260" watchObservedRunningTime="2026-09-16 12:46:43.060195034 +0000 UTC m=+509.507427675"
Sep 16 12:46:46.632637 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:46.632596    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c7bc95455-h5zdt"]
Sep 16 12:46:46.636335 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:46.636313    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-h5zdt"
Sep 16 12:46:46.639900 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:46:46.639850    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-manager-service-cert\" is forbidden: User \"system:node:ip-10-0-133-190.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-190.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret"
Sep 16 12:46:46.640080 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:46:46.639896    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"opendatahub-operator-controller-manager-5c7bc95455-h5zdt\" is forbidden: User \"system:node:ip-10-0-133-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-190.ec2.internal' and this object" podUID="ad54ab6a-787b-471a-a648-1badf8c1b94c" pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-h5zdt"
Sep 16 12:46:46.640080 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:46:46.639965    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-133-190.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-190.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Sep 16 12:46:46.640619 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:46:46.640593    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-133-190.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-190.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Sep 16 12:46:46.640709 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:46:46.640636    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-manager-dockercfg-72j5p\" is forbidden: User \"system:node:ip-10-0-133-190.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-190.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-72j5p\"" type="*v1.Secret"
Sep 16 12:46:46.640709 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:46:46.640636    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-webhook-cert\" is forbidden: User \"system:node:ip-10-0-133-190.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-190.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" type="*v1.Secret"
Sep 16 12:46:46.655181 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:46.655158    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c7bc95455-h5zdt"]
Sep 16 12:46:46.701434 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:46.701407    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ad54ab6a-787b-471a-a648-1badf8c1b94c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c7bc95455-h5zdt\" (UID: \"ad54ab6a-787b-471a-a648-1badf8c1b94c\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-h5zdt"
Sep 16 12:46:46.701557 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:46.701439    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rzxr\" (UniqueName: \"kubernetes.io/projected/ad54ab6a-787b-471a-a648-1badf8c1b94c-kube-api-access-4rzxr\") pod \"opendatahub-operator-controller-manager-5c7bc95455-h5zdt\" (UID: \"ad54ab6a-787b-471a-a648-1badf8c1b94c\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-h5zdt"
Sep 16 12:46:46.701557 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:46.701462    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ad54ab6a-787b-471a-a648-1badf8c1b94c-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c7bc95455-h5zdt\" (UID: \"ad54ab6a-787b-471a-a648-1badf8c1b94c\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-h5zdt"
Sep 16 12:46:46.701662 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:46.701565    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad54ab6a-787b-471a-a648-1badf8c1b94c-tmp\") pod \"opendatahub-operator-controller-manager-5c7bc95455-h5zdt\" (UID: \"ad54ab6a-787b-471a-a648-1badf8c1b94c\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-h5zdt"
Sep 16 12:46:46.701662 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:46.701614    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ad54ab6a-787b-471a-a648-1badf8c1b94c-manifests\") pod \"opendatahub-operator-controller-manager-5c7bc95455-h5zdt\" (UID: \"ad54ab6a-787b-471a-a648-1badf8c1b94c\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-h5zdt"
Sep 16 12:46:46.802775 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:46.802742    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ad54ab6a-787b-471a-a648-1badf8c1b94c-manifests\") pod \"opendatahub-operator-controller-manager-5c7bc95455-h5zdt\" (UID: \"ad54ab6a-787b-471a-a648-1badf8c1b94c\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-h5zdt"
Sep 16 12:46:46.802945 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:46.802796    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ad54ab6a-787b-471a-a648-1badf8c1b94c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c7bc95455-h5zdt\" (UID: \"ad54ab6a-787b-471a-a648-1badf8c1b94c\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-h5zdt"
Sep 16 12:46:46.802945 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:46.802813    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rzxr\" (UniqueName: \"kubernetes.io/projected/ad54ab6a-787b-471a-a648-1badf8c1b94c-kube-api-access-4rzxr\") pod \"opendatahub-operator-controller-manager-5c7bc95455-h5zdt\" (UID: \"ad54ab6a-787b-471a-a648-1badf8c1b94c\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-h5zdt"
Sep 16 12:46:46.802945 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:46.802833    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ad54ab6a-787b-471a-a648-1badf8c1b94c-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c7bc95455-h5zdt\" (UID: \"ad54ab6a-787b-471a-a648-1badf8c1b94c\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-h5zdt"
Sep 16 12:46:46.802945 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:46.802871    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad54ab6a-787b-471a-a648-1badf8c1b94c-tmp\") pod \"opendatahub-operator-controller-manager-5c7bc95455-h5zdt\" (UID: \"ad54ab6a-787b-471a-a648-1badf8c1b94c\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-h5zdt"
Sep 16 12:46:46.803188 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:46.803170    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ad54ab6a-787b-471a-a648-1badf8c1b94c-manifests\") pod \"opendatahub-operator-controller-manager-5c7bc95455-h5zdt\" (UID: \"ad54ab6a-787b-471a-a648-1badf8c1b94c\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-h5zdt"
Sep 16 12:46:46.805134 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:46.805111    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad54ab6a-787b-471a-a648-1badf8c1b94c-tmp\") pod \"opendatahub-operator-controller-manager-5c7bc95455-h5zdt\" (UID: \"ad54ab6a-787b-471a-a648-1badf8c1b94c\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-h5zdt"
Sep 16 12:46:47.656429 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:47.656399    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 16 12:46:47.722426 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:47.722395    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 16 12:46:47.725853 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:47.725827    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ad54ab6a-787b-471a-a648-1badf8c1b94c-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c7bc95455-h5zdt\" (UID: \"ad54ab6a-787b-471a-a648-1badf8c1b94c\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-h5zdt"
Sep 16 12:46:47.725990 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:47.725921    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ad54ab6a-787b-471a-a648-1badf8c1b94c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c7bc95455-h5zdt\" (UID: \"ad54ab6a-787b-471a-a648-1badf8c1b94c\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-h5zdt"
Sep 16 12:46:47.812615 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:46:47.812588    2577 projected.go:291] Couldn't get configMap opendatahub/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition
Sep 16 12:46:47.822618 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:47.822594    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-72j5p\""
Sep 16 12:46:47.876398 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:47.876375    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 16 12:46:48.023019 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:48.022993    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 16 12:46:48.033269 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:46:48.033243    2577 projected.go:196] Error preparing data for projected volume kube-api-access-4rzxr for pod opendatahub/opendatahub-operator-controller-manager-5c7bc95455-h5zdt: failed to sync configmap cache: timed out waiting for the condition
Sep 16 12:46:48.033397 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:46:48.033324    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad54ab6a-787b-471a-a648-1badf8c1b94c-kube-api-access-4rzxr podName:ad54ab6a-787b-471a-a648-1badf8c1b94c nodeName:}" failed. No retries permitted until 2026-09-16 12:46:48.533306687 +0000 UTC m=+514.980539310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4rzxr" (UniqueName: "kubernetes.io/projected/ad54ab6a-787b-471a-a648-1badf8c1b94c-kube-api-access-4rzxr") pod "opendatahub-operator-controller-manager-5c7bc95455-h5zdt" (UID: "ad54ab6a-787b-471a-a648-1badf8c1b94c") : failed to sync configmap cache: timed out waiting for the condition
Sep 16 12:46:48.620243 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:48.620212    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rzxr\" (UniqueName: \"kubernetes.io/projected/ad54ab6a-787b-471a-a648-1badf8c1b94c-kube-api-access-4rzxr\") pod \"opendatahub-operator-controller-manager-5c7bc95455-h5zdt\" (UID: \"ad54ab6a-787b-471a-a648-1badf8c1b94c\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-h5zdt"
Sep 16 12:46:48.622806 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:48.622777    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rzxr\" (UniqueName: \"kubernetes.io/projected/ad54ab6a-787b-471a-a648-1badf8c1b94c-kube-api-access-4rzxr\") pod \"opendatahub-operator-controller-manager-5c7bc95455-h5zdt\" (UID: \"ad54ab6a-787b-471a-a648-1badf8c1b94c\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-h5zdt"
Sep 16 12:46:48.747264 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:48.747228    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-h5zdt"
Sep 16 12:46:49.027513 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:49.027487    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c7bc95455-h5zdt"]
Sep 16 12:46:49.028929 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:46:49.028900    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad54ab6a_787b_471a_a648_1badf8c1b94c.slice/crio-6104c12ad408019f5b576dc32a840f909f8d719034a23cbc8cea59d677603b9f WatchSource:0}: Error finding container 6104c12ad408019f5b576dc32a840f909f8d719034a23cbc8cea59d677603b9f: Status 404 returned error can't find the container with id 6104c12ad408019f5b576dc32a840f909f8d719034a23cbc8cea59d677603b9f
Sep 16 12:46:49.057047 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:49.056997    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-h5zdt" event={"ID":"ad54ab6a-787b-471a-a648-1badf8c1b94c","Type":"ContainerStarted","Data":"6104c12ad408019f5b576dc32a840f909f8d719034a23cbc8cea59d677603b9f"}
Sep 16 12:46:51.035706 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:51.035658    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-689cccd495-wnwvg" podUID="2a290979-8aa8-45cc-b7cd-4942036b2c5b" containerName="console" containerID="cri-o://8b338e7288a18980424d17c2e100e37a2f1bc6edf2ed8ef8d7249a486b4442fb" gracePeriod=15
Sep 16 12:46:52.075611 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:52.075564    2577 generic.go:356] "Generic (PLEG): container finished" podID="2a290979-8aa8-45cc-b7cd-4942036b2c5b" containerID="8b338e7288a18980424d17c2e100e37a2f1bc6edf2ed8ef8d7249a486b4442fb" exitCode=0
Sep 16 12:46:52.075971 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:52.075763    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-689cccd495-wnwvg" event={"ID":"2a290979-8aa8-45cc-b7cd-4942036b2c5b","Type":"ContainerDied","Data":"8b338e7288a18980424d17c2e100e37a2f1bc6edf2ed8ef8d7249a486b4442fb"}
Sep 16 12:46:52.140658 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:52.140636    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-689cccd495-wnwvg"
Sep 16 12:46:52.252828 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:52.252744    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a290979-8aa8-45cc-b7cd-4942036b2c5b-oauth-serving-cert\") pod \"2a290979-8aa8-45cc-b7cd-4942036b2c5b\" (UID: \"2a290979-8aa8-45cc-b7cd-4942036b2c5b\") "
Sep 16 12:46:52.252828 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:52.252802    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a290979-8aa8-45cc-b7cd-4942036b2c5b-trusted-ca-bundle\") pod \"2a290979-8aa8-45cc-b7cd-4942036b2c5b\" (UID: \"2a290979-8aa8-45cc-b7cd-4942036b2c5b\") "
Sep 16 12:46:52.253089 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:52.252835    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a290979-8aa8-45cc-b7cd-4942036b2c5b-service-ca\") pod \"2a290979-8aa8-45cc-b7cd-4942036b2c5b\" (UID: \"2a290979-8aa8-45cc-b7cd-4942036b2c5b\") "
Sep 16 12:46:52.253089 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:52.252864    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkd5h\" (UniqueName: \"kubernetes.io/projected/2a290979-8aa8-45cc-b7cd-4942036b2c5b-kube-api-access-xkd5h\") pod \"2a290979-8aa8-45cc-b7cd-4942036b2c5b\" (UID: \"2a290979-8aa8-45cc-b7cd-4942036b2c5b\") "
Sep 16 12:46:52.253089 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:52.252910    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a290979-8aa8-45cc-b7cd-4942036b2c5b-console-oauth-config\") pod \"2a290979-8aa8-45cc-b7cd-4942036b2c5b\" (UID: \"2a290979-8aa8-45cc-b7cd-4942036b2c5b\") "
Sep 16 12:46:52.253089 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:52.252935    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a290979-8aa8-45cc-b7cd-4942036b2c5b-console-config\") pod \"2a290979-8aa8-45cc-b7cd-4942036b2c5b\" (UID: \"2a290979-8aa8-45cc-b7cd-4942036b2c5b\") "
Sep 16 12:46:52.253089 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:52.252966    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a290979-8aa8-45cc-b7cd-4942036b2c5b-console-serving-cert\") pod \"2a290979-8aa8-45cc-b7cd-4942036b2c5b\" (UID: \"2a290979-8aa8-45cc-b7cd-4942036b2c5b\") "
Sep 16 12:46:52.253356 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:52.253250    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a290979-8aa8-45cc-b7cd-4942036b2c5b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2a290979-8aa8-45cc-b7cd-4942036b2c5b" (UID: "2a290979-8aa8-45cc-b7cd-4942036b2c5b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:46:52.253356 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:52.253304    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a290979-8aa8-45cc-b7cd-4942036b2c5b-service-ca" (OuterVolumeSpecName: "service-ca") pod "2a290979-8aa8-45cc-b7cd-4942036b2c5b" (UID: "2a290979-8aa8-45cc-b7cd-4942036b2c5b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:46:52.253467 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:52.253353    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a290979-8aa8-45cc-b7cd-4942036b2c5b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2a290979-8aa8-45cc-b7cd-4942036b2c5b" (UID: "2a290979-8aa8-45cc-b7cd-4942036b2c5b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:46:52.253467 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:52.253363    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a290979-8aa8-45cc-b7cd-4942036b2c5b-console-config" (OuterVolumeSpecName: "console-config") pod "2a290979-8aa8-45cc-b7cd-4942036b2c5b" (UID: "2a290979-8aa8-45cc-b7cd-4942036b2c5b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:46:52.255169 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:52.255143    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a290979-8aa8-45cc-b7cd-4942036b2c5b-kube-api-access-xkd5h" (OuterVolumeSpecName: "kube-api-access-xkd5h") pod "2a290979-8aa8-45cc-b7cd-4942036b2c5b" (UID: "2a290979-8aa8-45cc-b7cd-4942036b2c5b"). InnerVolumeSpecName "kube-api-access-xkd5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:46:52.255296 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:52.255226    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a290979-8aa8-45cc-b7cd-4942036b2c5b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2a290979-8aa8-45cc-b7cd-4942036b2c5b" (UID: "2a290979-8aa8-45cc-b7cd-4942036b2c5b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:46:52.255296 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:52.255255    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a290979-8aa8-45cc-b7cd-4942036b2c5b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2a290979-8aa8-45cc-b7cd-4942036b2c5b" (UID: "2a290979-8aa8-45cc-b7cd-4942036b2c5b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:46:52.354053 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:52.354000    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkd5h\" (UniqueName: \"kubernetes.io/projected/2a290979-8aa8-45cc-b7cd-4942036b2c5b-kube-api-access-xkd5h\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:46:52.354053 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:52.354052    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a290979-8aa8-45cc-b7cd-4942036b2c5b-console-oauth-config\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:46:52.354238 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:52.354067    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a290979-8aa8-45cc-b7cd-4942036b2c5b-console-config\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:46:52.354238 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:52.354080    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a290979-8aa8-45cc-b7cd-4942036b2c5b-console-serving-cert\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:46:52.354238 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:52.354091    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a290979-8aa8-45cc-b7cd-4942036b2c5b-oauth-serving-cert\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:46:52.354238 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:52.354102    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a290979-8aa8-45cc-b7cd-4942036b2c5b-trusted-ca-bundle\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:46:52.354238 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:52.354113    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a290979-8aa8-45cc-b7cd-4942036b2c5b-service-ca\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:46:53.081664 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:53.081625    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-h5zdt" event={"ID":"ad54ab6a-787b-471a-a648-1badf8c1b94c","Type":"ContainerStarted","Data":"b690c52a2bd944961b731f5e53c2966b9e98a9e9b52a96b2cbf24117a2ad1857"}
Sep 16 12:46:53.082968 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:53.082951    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-689cccd495-wnwvg"
Sep 16 12:46:53.083070 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:53.082953    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-689cccd495-wnwvg" event={"ID":"2a290979-8aa8-45cc-b7cd-4942036b2c5b","Type":"ContainerDied","Data":"cecbf66037ee59af5bd5662d8dc470429d7528dafb2bf21fdef263ef9f700e29"}
Sep 16 12:46:53.083119 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:53.083085    2577 scope.go:117] "RemoveContainer" containerID="8b338e7288a18980424d17c2e100e37a2f1bc6edf2ed8ef8d7249a486b4442fb"
Sep 16 12:46:53.119765 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:53.119743    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-689cccd495-wnwvg"]
Sep 16 12:46:53.125611 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:53.125590    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-689cccd495-wnwvg"]
Sep 16 12:46:54.037179 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:54.037153    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-575fbb86f4-qr6g8"
Sep 16 12:46:54.054418 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:54.054390    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a290979-8aa8-45cc-b7cd-4942036b2c5b" path="/var/lib/kubelet/pods/2a290979-8aa8-45cc-b7cd-4942036b2c5b/volumes"
Sep 16 12:46:54.091735 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:54.091692    2577 generic.go:356] "Generic (PLEG): container finished" podID="ad54ab6a-787b-471a-a648-1badf8c1b94c" containerID="b690c52a2bd944961b731f5e53c2966b9e98a9e9b52a96b2cbf24117a2ad1857" exitCode=0
Sep 16 12:46:54.092102 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:54.091758    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-h5zdt" event={"ID":"ad54ab6a-787b-471a-a648-1badf8c1b94c","Type":"ContainerDied","Data":"b690c52a2bd944961b731f5e53c2966b9e98a9e9b52a96b2cbf24117a2ad1857"}
Sep 16 12:46:55.097566 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:55.097531    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-h5zdt" event={"ID":"ad54ab6a-787b-471a-a648-1badf8c1b94c","Type":"ContainerStarted","Data":"2cf7ba03bdfefc681ad631457329c9171beada8139a85e67c1f9116001fc9b20"}
Sep 16 12:46:55.097948 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:55.097650    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-h5zdt"
Sep 16 12:46:55.124127 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:46:55.124080    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-h5zdt" podStartSLOduration=4.6403333369999995 podStartE2EDuration="9.124065569s" totalImagesPullingTime="3.413506646s" totalInitContainerRuntime="1.070225586s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:46:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:46:55.122446595 +0000 UTC m=+521.569679248" watchObservedRunningTime="2026-09-16 12:46:55.124065569 +0000 UTC m=+521.571298210"
Sep 16 12:47:06.103324 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:06.103255    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-h5zdt"
Sep 16 12:47:27.316069 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.316019    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm"]
Sep 16 12:47:27.316443 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.316354    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a290979-8aa8-45cc-b7cd-4942036b2c5b" containerName="console"
Sep 16 12:47:27.316443 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.316365    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a290979-8aa8-45cc-b7cd-4942036b2c5b" containerName="console"
Sep 16 12:47:27.316443 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.316421    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a290979-8aa8-45cc-b7cd-4942036b2c5b" containerName="console"
Sep 16 12:47:27.320746 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.320721    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm"
Sep 16 12:47:27.325103 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.325083    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 12:47:27.325276 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.325260    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-5vg4r\""
Sep 16 12:47:27.379447 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.379414    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm"]
Sep 16 12:47:27.383148 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.383115    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s"]
Sep 16 12:47:27.386448 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.386425    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s"
Sep 16 12:47:27.403719 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.403696    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s"]
Sep 16 12:47:27.450267 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.450239    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5004e84-f83f-4c62-b8f6-afff582db754-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm\" (UID: \"e5004e84-f83f-4c62-b8f6-afff582db754\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm"
Sep 16 12:47:27.450402 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.450271    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5004e84-f83f-4c62-b8f6-afff582db754-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm\" (UID: \"e5004e84-f83f-4c62-b8f6-afff582db754\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm"
Sep 16 12:47:27.450402 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.450299    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5004e84-f83f-4c62-b8f6-afff582db754-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm\" (UID: \"e5004e84-f83f-4c62-b8f6-afff582db754\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm"
Sep 16 12:47:27.450402 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.450330    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5004e84-f83f-4c62-b8f6-afff582db754-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm\" (UID: \"e5004e84-f83f-4c62-b8f6-afff582db754\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm"
Sep 16 12:47:27.450402 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.450349    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef53d612-0d2e-426f-a926-bcee1b7d4827-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s\" (UID: \"ef53d612-0d2e-426f-a926-bcee1b7d4827\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s"
Sep 16 12:47:27.450402 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.450366    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5004e84-f83f-4c62-b8f6-afff582db754-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm\" (UID: \"e5004e84-f83f-4c62-b8f6-afff582db754\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm"
Sep 16 12:47:27.450402 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.450382    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef53d612-0d2e-426f-a926-bcee1b7d4827-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s\" (UID: \"ef53d612-0d2e-426f-a926-bcee1b7d4827\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s"
Sep 16 12:47:27.450693 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.450439    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqlx8\" (UniqueName: \"kubernetes.io/projected/e5004e84-f83f-4c62-b8f6-afff582db754-kube-api-access-rqlx8\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm\" (UID: \"e5004e84-f83f-4c62-b8f6-afff582db754\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm"
Sep 16 12:47:27.450693 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.450482    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-692w2\" (UniqueName: \"kubernetes.io/projected/ef53d612-0d2e-426f-a926-bcee1b7d4827-kube-api-access-692w2\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s\" (UID: \"ef53d612-0d2e-426f-a926-bcee1b7d4827\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s"
Sep 16 12:47:27.450693 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.450522    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5004e84-f83f-4c62-b8f6-afff582db754-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm\" (UID: \"e5004e84-f83f-4c62-b8f6-afff582db754\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm"
Sep 16 12:47:27.450693 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.450544    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef53d612-0d2e-426f-a926-bcee1b7d4827-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s\" (UID: \"ef53d612-0d2e-426f-a926-bcee1b7d4827\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s"
Sep 16 12:47:27.450693 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.450569    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef53d612-0d2e-426f-a926-bcee1b7d4827-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s\" (UID: \"ef53d612-0d2e-426f-a926-bcee1b7d4827\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s"
Sep 16 12:47:27.450693 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.450601    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef53d612-0d2e-426f-a926-bcee1b7d4827-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s\" (UID: \"ef53d612-0d2e-426f-a926-bcee1b7d4827\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s"
Sep 16 12:47:27.450693 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.450623    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef53d612-0d2e-426f-a926-bcee1b7d4827-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s\" (UID: \"ef53d612-0d2e-426f-a926-bcee1b7d4827\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s"
Sep 16 12:47:27.450693 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.450654    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5004e84-f83f-4c62-b8f6-afff582db754-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm\" (UID: \"e5004e84-f83f-4c62-b8f6-afff582db754\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm"
Sep 16 12:47:27.450693 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.450670    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef53d612-0d2e-426f-a926-bcee1b7d4827-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s\" (UID: \"ef53d612-0d2e-426f-a926-bcee1b7d4827\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s"
Sep 16 12:47:27.450693 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.450691    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5004e84-f83f-4c62-b8f6-afff582db754-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm\" (UID: \"e5004e84-f83f-4c62-b8f6-afff582db754\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm"
Sep 16 12:47:27.451011 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.450711    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef53d612-0d2e-426f-a926-bcee1b7d4827-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s\" (UID: \"ef53d612-0d2e-426f-a926-bcee1b7d4827\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s"
Sep 16 12:47:27.551724 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.551695    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5004e84-f83f-4c62-b8f6-afff582db754-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm\" (UID: \"e5004e84-f83f-4c62-b8f6-afff582db754\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm"
Sep 16 12:47:27.551724 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.551727    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5004e84-f83f-4c62-b8f6-afff582db754-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm\" (UID: \"e5004e84-f83f-4c62-b8f6-afff582db754\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm"
Sep 16 12:47:27.551966 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.551752    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5004e84-f83f-4c62-b8f6-afff582db754-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm\" (UID: \"e5004e84-f83f-4c62-b8f6-afff582db754\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm"
Sep 16 12:47:27.551966 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.551786    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5004e84-f83f-4c62-b8f6-afff582db754-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm\" (UID: \"e5004e84-f83f-4c62-b8f6-afff582db754\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm"
Sep 16 12:47:27.551966 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.551814    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef53d612-0d2e-426f-a926-bcee1b7d4827-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s\" (UID: \"ef53d612-0d2e-426f-a926-bcee1b7d4827\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s"
Sep 16 12:47:27.551966 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.551840    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5004e84-f83f-4c62-b8f6-afff582db754-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm\" (UID: \"e5004e84-f83f-4c62-b8f6-afff582db754\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm"
Sep 16 12:47:27.551966 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.551865    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef53d612-0d2e-426f-a926-bcee1b7d4827-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s\" (UID: \"ef53d612-0d2e-426f-a926-bcee1b7d4827\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s"
Sep 16 12:47:27.551966 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.551894    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqlx8\" (UniqueName: \"kubernetes.io/projected/e5004e84-f83f-4c62-b8f6-afff582db754-kube-api-access-rqlx8\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm\" (UID: \"e5004e84-f83f-4c62-b8f6-afff582db754\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm"
Sep 16 12:47:27.551966 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.551933    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-692w2\" (UniqueName: \"kubernetes.io/projected/ef53d612-0d2e-426f-a926-bcee1b7d4827-kube-api-access-692w2\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s\" (UID: \"ef53d612-0d2e-426f-a926-bcee1b7d4827\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s"
Sep 16 12:47:27.552346 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.552001    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5004e84-f83f-4c62-b8f6-afff582db754-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm\" (UID: \"e5004e84-f83f-4c62-b8f6-afff582db754\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm"
Sep 16 12:47:27.552346 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.552062    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef53d612-0d2e-426f-a926-bcee1b7d4827-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s\" (UID: \"ef53d612-0d2e-426f-a926-bcee1b7d4827\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s"
Sep 16 12:47:27.552346 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.552090    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef53d612-0d2e-426f-a926-bcee1b7d4827-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s\" (UID: \"ef53d612-0d2e-426f-a926-bcee1b7d4827\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s"
Sep 16 12:47:27.552346 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.552143    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef53d612-0d2e-426f-a926-bcee1b7d4827-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s\" (UID: \"ef53d612-0d2e-426f-a926-bcee1b7d4827\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s"
Sep 16 12:47:27.552346 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.552173    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef53d612-0d2e-426f-a926-bcee1b7d4827-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s\" (UID: \"ef53d612-0d2e-426f-a926-bcee1b7d4827\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s"
Sep 16 12:47:27.552346 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.552202    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5004e84-f83f-4c62-b8f6-afff582db754-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm\" (UID: \"e5004e84-f83f-4c62-b8f6-afff582db754\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm"
Sep 16 12:47:27.552346 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.552230    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef53d612-0d2e-426f-a926-bcee1b7d4827-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s\" (UID: \"ef53d612-0d2e-426f-a926-bcee1b7d4827\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s"
Sep 16 12:47:27.552346 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.552237    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5004e84-f83f-4c62-b8f6-afff582db754-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm\" (UID: \"e5004e84-f83f-4c62-b8f6-afff582db754\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm"
Sep 16 12:47:27.552346 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.552254    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5004e84-f83f-4c62-b8f6-afff582db754-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm\" (UID: \"e5004e84-f83f-4c62-b8f6-afff582db754\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm"
Sep 16 12:47:27.552346 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.552260    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5004e84-f83f-4c62-b8f6-afff582db754-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm\" (UID: \"e5004e84-f83f-4c62-b8f6-afff582db754\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm"
Sep 16 12:47:27.552346 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.552332    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef53d612-0d2e-426f-a926-bcee1b7d4827-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s\" (UID: \"ef53d612-0d2e-426f-a926-bcee1b7d4827\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s"
Sep 16 12:47:27.553051 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.552700    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef53d612-0d2e-426f-a926-bcee1b7d4827-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s\" (UID: \"ef53d612-0d2e-426f-a926-bcee1b7d4827\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s"
Sep 16 12:47:27.553051 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.552903    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5004e84-f83f-4c62-b8f6-afff582db754-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm\" (UID: \"e5004e84-f83f-4c62-b8f6-afff582db754\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm"
Sep 16 12:47:27.553051 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.552926    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5004e84-f83f-4c62-b8f6-afff582db754-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm\" (UID: \"e5004e84-f83f-4c62-b8f6-afff582db754\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm"
Sep 16 12:47:27.553051 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.552982    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef53d612-0d2e-426f-a926-bcee1b7d4827-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s\" (UID: \"ef53d612-0d2e-426f-a926-bcee1b7d4827\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s"
Sep 16 12:47:27.553265 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.553137    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef53d612-0d2e-426f-a926-bcee1b7d4827-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s\" (UID: \"ef53d612-0d2e-426f-a926-bcee1b7d4827\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s"
Sep 16 12:47:27.553265 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.553201    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef53d612-0d2e-426f-a926-bcee1b7d4827-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s\" (UID: \"ef53d612-0d2e-426f-a926-bcee1b7d4827\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s"
Sep 16 12:47:27.553394 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.553370    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef53d612-0d2e-426f-a926-bcee1b7d4827-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s\" (UID: \"ef53d612-0d2e-426f-a926-bcee1b7d4827\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s"
Sep 16 12:47:27.553450 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.553398    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5004e84-f83f-4c62-b8f6-afff582db754-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm\" (UID: \"e5004e84-f83f-4c62-b8f6-afff582db754\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm"
Sep 16 12:47:27.554727 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.554704    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef53d612-0d2e-426f-a926-bcee1b7d4827-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s\" (UID: \"ef53d612-0d2e-426f-a926-bcee1b7d4827\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s"
Sep 16 12:47:27.554835 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.554745    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5004e84-f83f-4c62-b8f6-afff582db754-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm\" (UID: \"e5004e84-f83f-4c62-b8f6-afff582db754\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm"
Sep 16 12:47:27.554894 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.554874    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef53d612-0d2e-426f-a926-bcee1b7d4827-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s\" (UID: \"ef53d612-0d2e-426f-a926-bcee1b7d4827\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s"
Sep 16 12:47:27.555161 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.555144    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5004e84-f83f-4c62-b8f6-afff582db754-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm\" (UID: \"e5004e84-f83f-4c62-b8f6-afff582db754\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm"
Sep 16 12:47:27.566964 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.566866    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef53d612-0d2e-426f-a926-bcee1b7d4827-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s\" (UID: \"ef53d612-0d2e-426f-a926-bcee1b7d4827\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s"
Sep 16 12:47:27.567142 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.567125    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5004e84-f83f-4c62-b8f6-afff582db754-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm\" (UID: \"e5004e84-f83f-4c62-b8f6-afff582db754\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm"
Sep 16 12:47:27.567575 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.567551    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqlx8\" (UniqueName: \"kubernetes.io/projected/e5004e84-f83f-4c62-b8f6-afff582db754-kube-api-access-rqlx8\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm\" (UID: \"e5004e84-f83f-4c62-b8f6-afff582db754\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm"
Sep 16 12:47:27.567643 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.567573    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-692w2\" (UniqueName: \"kubernetes.io/projected/ef53d612-0d2e-426f-a926-bcee1b7d4827-kube-api-access-692w2\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s\" (UID: \"ef53d612-0d2e-426f-a926-bcee1b7d4827\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s"
Sep 16 12:47:27.631302 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.631274    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm"
Sep 16 12:47:27.696996 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.696387    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s"
Sep 16 12:47:27.919671 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:27.919642    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm"]
Sep 16 12:47:27.920538 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:47:27.920514    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5004e84_f83f_4c62_b8f6_afff582db754.slice/crio-3c4a48d3aac007146c32dd52c3475f86d752641727138dcd5a3669e6a882b2dd WatchSource:0}: Error finding container 3c4a48d3aac007146c32dd52c3475f86d752641727138dcd5a3669e6a882b2dd: Status 404 returned error can't find the container with id 3c4a48d3aac007146c32dd52c3475f86d752641727138dcd5a3669e6a882b2dd
Sep 16 12:47:28.002139 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:28.002117    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s"]
Sep 16 12:47:28.003993 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:47:28.003974    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef53d612_0d2e_426f_a926_bcee1b7d4827.slice/crio-fe02d5e0495bacdc166bf07810533d96ef5e4ca65cb1ef36afcc224d20131d3c WatchSource:0}: Error finding container fe02d5e0495bacdc166bf07810533d96ef5e4ca65cb1ef36afcc224d20131d3c: Status 404 returned error can't find the container with id fe02d5e0495bacdc166bf07810533d96ef5e4ca65cb1ef36afcc224d20131d3c
Sep 16 12:47:28.244892 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:28.244806    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s" event={"ID":"ef53d612-0d2e-426f-a926-bcee1b7d4827","Type":"ContainerStarted","Data":"fe02d5e0495bacdc166bf07810533d96ef5e4ca65cb1ef36afcc224d20131d3c"}
Sep 16 12:47:28.245988 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:28.245963    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm" event={"ID":"e5004e84-f83f-4c62-b8f6-afff582db754","Type":"ContainerStarted","Data":"3c4a48d3aac007146c32dd52c3475f86d752641727138dcd5a3669e6a882b2dd"}
Sep 16 12:47:30.969313 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:30.969273    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-ssfwx"]
Sep 16 12:47:30.972774 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:30.972756    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ssfwx"
Sep 16 12:47:30.976763 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:30.976738    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-grwjs\""
Sep 16 12:47:30.977421 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:30.977390    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 16 12:47:30.984960 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:30.984914    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:47:30.985074 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:30.984992    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:47:30.985074 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:30.985021    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:47:30.992754 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:30.992725    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:47:30.992902 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:30.992796    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:47:30.992902 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:30.992840    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:47:31.000472 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:31.000449    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-ssfwx"]
Sep 16 12:47:31.090883 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:31.090860    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7dbj\" (UniqueName: \"kubernetes.io/projected/9ed52a74-3e4e-4519-b070-f1bca5653647-kube-api-access-l7dbj\") pod \"llmisvc-controller-manager-777c6bd7d5-ssfwx\" (UID: \"9ed52a74-3e4e-4519-b070-f1bca5653647\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ssfwx"
Sep 16 12:47:31.091099 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:31.091079    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ed52a74-3e4e-4519-b070-f1bca5653647-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-ssfwx\" (UID: \"9ed52a74-3e4e-4519-b070-f1bca5653647\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ssfwx"
Sep 16 12:47:31.192131 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:31.192096    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7dbj\" (UniqueName: \"kubernetes.io/projected/9ed52a74-3e4e-4519-b070-f1bca5653647-kube-api-access-l7dbj\") pod \"llmisvc-controller-manager-777c6bd7d5-ssfwx\" (UID: \"9ed52a74-3e4e-4519-b070-f1bca5653647\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ssfwx"
Sep 16 12:47:31.192296 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:31.192153    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ed52a74-3e4e-4519-b070-f1bca5653647-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-ssfwx\" (UID: \"9ed52a74-3e4e-4519-b070-f1bca5653647\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ssfwx"
Sep 16 12:47:31.194843 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:31.194789    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ed52a74-3e4e-4519-b070-f1bca5653647-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-ssfwx\" (UID: \"9ed52a74-3e4e-4519-b070-f1bca5653647\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ssfwx"
Sep 16 12:47:31.207279 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:31.207062    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7dbj\" (UniqueName: \"kubernetes.io/projected/9ed52a74-3e4e-4519-b070-f1bca5653647-kube-api-access-l7dbj\") pod \"llmisvc-controller-manager-777c6bd7d5-ssfwx\" (UID: \"9ed52a74-3e4e-4519-b070-f1bca5653647\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ssfwx"
Sep 16 12:47:31.263557 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:31.263521    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s" event={"ID":"ef53d612-0d2e-426f-a926-bcee1b7d4827","Type":"ContainerStarted","Data":"4be669f03983035d0aa1ca898596aab03a1d194d1d43e743e3d3843b786e9e53"}
Sep 16 12:47:31.265259 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:31.265232    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm" event={"ID":"e5004e84-f83f-4c62-b8f6-afff582db754","Type":"ContainerStarted","Data":"38d41bd6d77e92c17a61ea86acfebf859be9e31b3422fc9bc4a9c93fec3cfcdc"}
Sep 16 12:47:31.284987 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:31.284963    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ssfwx"
Sep 16 12:47:31.299641 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:31.299599    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s" podStartSLOduration=1.3209623160000001 podStartE2EDuration="4.299584025s" totalImagesPullingTime="2.978621709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:47:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:47:31.298668893 +0000 UTC m=+557.745901536" watchObservedRunningTime="2026-09-16 12:47:31.299584025 +0000 UTC m=+557.746816667"
Sep 16 12:47:31.327066 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:31.326960    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm" podStartSLOduration=1.257088669 podStartE2EDuration="4.326941743s" totalImagesPullingTime="3.069853074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:47:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:47:31.326585019 +0000 UTC m=+557.773817665" watchObservedRunningTime="2026-09-16 12:47:31.326941743 +0000 UTC m=+557.774174387"
Sep 16 12:47:31.586783 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:31.586730    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-ssfwx"]
Sep 16 12:47:31.588880 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:47:31.588851    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9ed52a74_3e4e_4519_b070_f1bca5653647.slice/crio-5b4d70dd66bb6731123bb43ff1dea9761cb2a56ccb7731afc637cd29c3943478 WatchSource:0}: Error finding container 5b4d70dd66bb6731123bb43ff1dea9761cb2a56ccb7731afc637cd29c3943478: Status 404 returned error can't find the container with id 5b4d70dd66bb6731123bb43ff1dea9761cb2a56ccb7731afc637cd29c3943478
Sep 16 12:47:31.632185 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:31.632161    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm"
Sep 16 12:47:31.696481 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:31.696450    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s"
Sep 16 12:47:32.272387 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:32.272357    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ssfwx" event={"ID":"9ed52a74-3e4e-4519-b070-f1bca5653647","Type":"ContainerStarted","Data":"5b4d70dd66bb6731123bb43ff1dea9761cb2a56ccb7731afc637cd29c3943478"}
Sep 16 12:47:32.633140 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:32.633063    2577 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" start-of-body=
Sep 16 12:47:32.633279 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:32.633127    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm" podUID="e5004e84-f83f-4c62-b8f6-afff582db754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused"
Sep 16 12:47:32.702409 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:32.702379    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s"
Sep 16 12:47:33.277629 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:33.277596    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s"
Sep 16 12:47:33.278864 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:33.278842    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s"
Sep 16 12:47:33.347457 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:33.347421    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm"]
Sep 16 12:47:33.632360 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:33.632277    2577 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" start-of-body=
Sep 16 12:47:33.632360 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:33.632345    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm" podUID="e5004e84-f83f-4c62-b8f6-afff582db754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused"
Sep 16 12:47:34.282794 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:34.282754    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm" podUID="e5004e84-f83f-4c62-b8f6-afff582db754" containerName="istio-proxy" containerID="cri-o://38d41bd6d77e92c17a61ea86acfebf859be9e31b3422fc9bc4a9c93fec3cfcdc" gracePeriod=30
Sep 16 12:47:35.289236 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:35.289202    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ssfwx" event={"ID":"9ed52a74-3e4e-4519-b070-f1bca5653647","Type":"ContainerStarted","Data":"5e946eef314dbea99b0d49b5485e5232ca8c5809d7be86548252598f1ce83f52"}
Sep 16 12:47:35.289666 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:35.289390    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ssfwx"
Sep 16 12:47:35.313389 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:35.313346    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ssfwx" podStartSLOduration=2.625773465 podStartE2EDuration="5.313330904s" totalImagesPullingTime="2.687557439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:47:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:47:35.312197089 +0000 UTC m=+561.759429730" watchObservedRunningTime="2026-09-16 12:47:35.313330904 +0000 UTC m=+561.760563604"
Sep 16 12:47:39.520671 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:39.520642    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm"
Sep 16 12:47:39.666106 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:39.666002    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5004e84-f83f-4c62-b8f6-afff582db754-istiod-ca-cert\") pod \"e5004e84-f83f-4c62-b8f6-afff582db754\" (UID: \"e5004e84-f83f-4c62-b8f6-afff582db754\") "
Sep 16 12:47:39.666106 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:39.666067    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5004e84-f83f-4c62-b8f6-afff582db754-credential-socket\") pod \"e5004e84-f83f-4c62-b8f6-afff582db754\" (UID: \"e5004e84-f83f-4c62-b8f6-afff582db754\") "
Sep 16 12:47:39.666106 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:39.666104    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5004e84-f83f-4c62-b8f6-afff582db754-istio-envoy\") pod \"e5004e84-f83f-4c62-b8f6-afff582db754\" (UID: \"e5004e84-f83f-4c62-b8f6-afff582db754\") "
Sep 16 12:47:39.666379 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:39.666121    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqlx8\" (UniqueName: \"kubernetes.io/projected/e5004e84-f83f-4c62-b8f6-afff582db754-kube-api-access-rqlx8\") pod \"e5004e84-f83f-4c62-b8f6-afff582db754\" (UID: \"e5004e84-f83f-4c62-b8f6-afff582db754\") "
Sep 16 12:47:39.666379 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:39.666182    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5004e84-f83f-4c62-b8f6-afff582db754-workload-certs\") pod \"e5004e84-f83f-4c62-b8f6-afff582db754\" (UID: \"e5004e84-f83f-4c62-b8f6-afff582db754\") "
Sep 16 12:47:39.666379 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:39.666212    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5004e84-f83f-4c62-b8f6-afff582db754-istio-podinfo\") pod \"e5004e84-f83f-4c62-b8f6-afff582db754\" (UID: \"e5004e84-f83f-4c62-b8f6-afff582db754\") "
Sep 16 12:47:39.666379 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:39.666249    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5004e84-f83f-4c62-b8f6-afff582db754-workload-socket\") pod \"e5004e84-f83f-4c62-b8f6-afff582db754\" (UID: \"e5004e84-f83f-4c62-b8f6-afff582db754\") "
Sep 16 12:47:39.666379 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:39.666275    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5004e84-f83f-4c62-b8f6-afff582db754-istio-token\") pod \"e5004e84-f83f-4c62-b8f6-afff582db754\" (UID: \"e5004e84-f83f-4c62-b8f6-afff582db754\") "
Sep 16 12:47:39.666379 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:39.666303    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5004e84-f83f-4c62-b8f6-afff582db754-istio-data\") pod \"e5004e84-f83f-4c62-b8f6-afff582db754\" (UID: \"e5004e84-f83f-4c62-b8f6-afff582db754\") "
Sep 16 12:47:39.666683 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:39.666391    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5004e84-f83f-4c62-b8f6-afff582db754-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e5004e84-f83f-4c62-b8f6-afff582db754" (UID: "e5004e84-f83f-4c62-b8f6-afff582db754"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:47:39.666683 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:39.666586    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5004e84-f83f-4c62-b8f6-afff582db754-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e5004e84-f83f-4c62-b8f6-afff582db754" (UID: "e5004e84-f83f-4c62-b8f6-afff582db754"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:47:39.666683 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:39.666619    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5004e84-f83f-4c62-b8f6-afff582db754-credential-socket\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:47:39.666683 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:39.666626    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5004e84-f83f-4c62-b8f6-afff582db754-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e5004e84-f83f-4c62-b8f6-afff582db754" (UID: "e5004e84-f83f-4c62-b8f6-afff582db754"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:47:39.666683 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:39.666604    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5004e84-f83f-4c62-b8f6-afff582db754-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e5004e84-f83f-4c62-b8f6-afff582db754" (UID: "e5004e84-f83f-4c62-b8f6-afff582db754"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:47:39.666683 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:39.666654    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5004e84-f83f-4c62-b8f6-afff582db754-istio-data" (OuterVolumeSpecName: "istio-data") pod "e5004e84-f83f-4c62-b8f6-afff582db754" (UID: "e5004e84-f83f-4c62-b8f6-afff582db754"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:47:39.668650 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:39.668623    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5004e84-f83f-4c62-b8f6-afff582db754-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e5004e84-f83f-4c62-b8f6-afff582db754" (UID: "e5004e84-f83f-4c62-b8f6-afff582db754"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:47:39.668650 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:39.668638    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5004e84-f83f-4c62-b8f6-afff582db754-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e5004e84-f83f-4c62-b8f6-afff582db754" (UID: "e5004e84-f83f-4c62-b8f6-afff582db754"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:47:39.668803 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:39.668678    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5004e84-f83f-4c62-b8f6-afff582db754-kube-api-access-rqlx8" (OuterVolumeSpecName: "kube-api-access-rqlx8") pod "e5004e84-f83f-4c62-b8f6-afff582db754" (UID: "e5004e84-f83f-4c62-b8f6-afff582db754"). InnerVolumeSpecName "kube-api-access-rqlx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:47:39.668869 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:39.668855    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5004e84-f83f-4c62-b8f6-afff582db754-istio-token" (OuterVolumeSpecName: "istio-token") pod "e5004e84-f83f-4c62-b8f6-afff582db754" (UID: "e5004e84-f83f-4c62-b8f6-afff582db754"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:47:39.767667 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:39.767639    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5004e84-f83f-4c62-b8f6-afff582db754-istiod-ca-cert\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:47:39.767667 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:39.767662    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5004e84-f83f-4c62-b8f6-afff582db754-istio-envoy\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:47:39.767667 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:39.767672    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqlx8\" (UniqueName: \"kubernetes.io/projected/e5004e84-f83f-4c62-b8f6-afff582db754-kube-api-access-rqlx8\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:47:39.767871 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:39.767681    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5004e84-f83f-4c62-b8f6-afff582db754-workload-certs\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:47:39.767871 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:39.767689    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5004e84-f83f-4c62-b8f6-afff582db754-istio-podinfo\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:47:39.767871 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:39.767697    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5004e84-f83f-4c62-b8f6-afff582db754-workload-socket\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:47:39.767871 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:39.767705    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5004e84-f83f-4c62-b8f6-afff582db754-istio-token\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:47:39.767871 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:39.767712    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5004e84-f83f-4c62-b8f6-afff582db754-istio-data\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:47:40.316764 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:40.316679    2577 generic.go:356] "Generic (PLEG): container finished" podID="e5004e84-f83f-4c62-b8f6-afff582db754" containerID="38d41bd6d77e92c17a61ea86acfebf859be9e31b3422fc9bc4a9c93fec3cfcdc" exitCode=0
Sep 16 12:47:40.316764 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:40.316725    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm"
Sep 16 12:47:40.316764 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:40.316729    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm" event={"ID":"e5004e84-f83f-4c62-b8f6-afff582db754","Type":"ContainerDied","Data":"38d41bd6d77e92c17a61ea86acfebf859be9e31b3422fc9bc4a9c93fec3cfcdc"}
Sep 16 12:47:40.316962 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:40.316780    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm" event={"ID":"e5004e84-f83f-4c62-b8f6-afff582db754","Type":"ContainerDied","Data":"3c4a48d3aac007146c32dd52c3475f86d752641727138dcd5a3669e6a882b2dd"}
Sep 16 12:47:40.316962 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:40.316806    2577 scope.go:117] "RemoveContainer" containerID="38d41bd6d77e92c17a61ea86acfebf859be9e31b3422fc9bc4a9c93fec3cfcdc"
Sep 16 12:47:40.325474 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:40.325451    2577 scope.go:117] "RemoveContainer" containerID="38d41bd6d77e92c17a61ea86acfebf859be9e31b3422fc9bc4a9c93fec3cfcdc"
Sep 16 12:47:40.325748 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:47:40.325727    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38d41bd6d77e92c17a61ea86acfebf859be9e31b3422fc9bc4a9c93fec3cfcdc\": container with ID starting with 38d41bd6d77e92c17a61ea86acfebf859be9e31b3422fc9bc4a9c93fec3cfcdc not found: ID does not exist" containerID="38d41bd6d77e92c17a61ea86acfebf859be9e31b3422fc9bc4a9c93fec3cfcdc"
Sep 16 12:47:40.325803 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:40.325759    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38d41bd6d77e92c17a61ea86acfebf859be9e31b3422fc9bc4a9c93fec3cfcdc"} err="failed to get container status \"38d41bd6d77e92c17a61ea86acfebf859be9e31b3422fc9bc4a9c93fec3cfcdc\": rpc error: code = NotFound desc = could not find container \"38d41bd6d77e92c17a61ea86acfebf859be9e31b3422fc9bc4a9c93fec3cfcdc\": container with ID starting with 38d41bd6d77e92c17a61ea86acfebf859be9e31b3422fc9bc4a9c93fec3cfcdc not found: ID does not exist"
Sep 16 12:47:40.339538 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:40.339513    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm"]
Sep 16 12:47:40.344188 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:40.344167    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfc4vm"]
Sep 16 12:47:42.054871 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:47:42.054835    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5004e84-f83f-4c62-b8f6-afff582db754" path="/var/lib/kubelet/pods/e5004e84-f83f-4c62-b8f6-afff582db754/volumes"
Sep 16 12:48:06.296144 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:06.296114    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ssfwx"
Sep 16 12:48:14.032856 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:14.032826    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vbt8d_0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174/console-operator/1.log"
Sep 16 12:48:14.033479 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:14.032956    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vbt8d_0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174/console-operator/1.log"
Sep 16 12:48:35.004377 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:35.004347    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-77dd6"]
Sep 16 12:48:35.004752 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:35.004697    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5004e84-f83f-4c62-b8f6-afff582db754" containerName="istio-proxy"
Sep 16 12:48:35.004752 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:35.004708    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5004e84-f83f-4c62-b8f6-afff582db754" containerName="istio-proxy"
Sep 16 12:48:35.004821 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:35.004764    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5004e84-f83f-4c62-b8f6-afff582db754" containerName="istio-proxy"
Sep 16 12:48:35.008868 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:35.008851    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-77dd6"
Sep 16 12:48:35.011742 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:35.011723    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 12:48:35.011871 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:35.011762    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 12:48:35.011871 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:35.011728    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 16 12:48:35.011871 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:35.011830    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-7fxv5\""
Sep 16 12:48:35.018148 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:35.018125    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-77dd6"]
Sep 16 12:48:35.132633 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:35.132555    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxbz8\" (UniqueName: \"kubernetes.io/projected/463edcc3-887a-4637-9702-bc62f152f9ff-kube-api-access-cxbz8\") pod \"dns-operator-controller-manager-7b84cd7c9c-77dd6\" (UID: \"463edcc3-887a-4637-9702-bc62f152f9ff\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-77dd6"
Sep 16 12:48:35.233213 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:35.233171    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxbz8\" (UniqueName: \"kubernetes.io/projected/463edcc3-887a-4637-9702-bc62f152f9ff-kube-api-access-cxbz8\") pod \"dns-operator-controller-manager-7b84cd7c9c-77dd6\" (UID: \"463edcc3-887a-4637-9702-bc62f152f9ff\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-77dd6"
Sep 16 12:48:35.254867 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:35.254802    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxbz8\" (UniqueName: \"kubernetes.io/projected/463edcc3-887a-4637-9702-bc62f152f9ff-kube-api-access-cxbz8\") pod \"dns-operator-controller-manager-7b84cd7c9c-77dd6\" (UID: \"463edcc3-887a-4637-9702-bc62f152f9ff\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-77dd6"
Sep 16 12:48:35.319161 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:35.319134    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-77dd6"
Sep 16 12:48:35.607660 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:35.607638    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-77dd6"]
Sep 16 12:48:35.610363 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:48:35.610329    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod463edcc3_887a_4637_9702_bc62f152f9ff.slice/crio-060279ce3d09fa7938201a1edcd551c6ae3defece7f70d532d318eb2090d6d0c WatchSource:0}: Error finding container 060279ce3d09fa7938201a1edcd551c6ae3defece7f70d532d318eb2090d6d0c: Status 404 returned error can't find the container with id 060279ce3d09fa7938201a1edcd551c6ae3defece7f70d532d318eb2090d6d0c
Sep 16 12:48:36.579504 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:36.579450    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-77dd6" event={"ID":"463edcc3-887a-4637-9702-bc62f152f9ff","Type":"ContainerStarted","Data":"060279ce3d09fa7938201a1edcd551c6ae3defece7f70d532d318eb2090d6d0c"}
Sep 16 12:48:37.756171 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:37.756133    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-wwn5q"]
Sep 16 12:48:37.760338 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:37.760317    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-wwn5q"
Sep 16 12:48:37.763264 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:37.763237    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-dht7t\""
Sep 16 12:48:37.772437 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:37.772414    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-wwn5q"]
Sep 16 12:48:37.856075 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:37.856020    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nzzw\" (UniqueName: \"kubernetes.io/projected/96b6599b-adb3-4b7c-b3fa-df54c38b0f00-kube-api-access-6nzzw\") pod \"authorino-operator-76b478bd78-wwn5q\" (UID: \"96b6599b-adb3-4b7c-b3fa-df54c38b0f00\") " pod="kuadrant-system/authorino-operator-76b478bd78-wwn5q"
Sep 16 12:48:37.957115 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:37.957070    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nzzw\" (UniqueName: \"kubernetes.io/projected/96b6599b-adb3-4b7c-b3fa-df54c38b0f00-kube-api-access-6nzzw\") pod \"authorino-operator-76b478bd78-wwn5q\" (UID: \"96b6599b-adb3-4b7c-b3fa-df54c38b0f00\") " pod="kuadrant-system/authorino-operator-76b478bd78-wwn5q"
Sep 16 12:48:37.967065 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:37.967039    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nzzw\" (UniqueName: \"kubernetes.io/projected/96b6599b-adb3-4b7c-b3fa-df54c38b0f00-kube-api-access-6nzzw\") pod \"authorino-operator-76b478bd78-wwn5q\" (UID: \"96b6599b-adb3-4b7c-b3fa-df54c38b0f00\") " pod="kuadrant-system/authorino-operator-76b478bd78-wwn5q"
Sep 16 12:48:38.073215 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:38.073184    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-wwn5q"
Sep 16 12:48:38.362159 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:38.361945    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-wwn5q"]
Sep 16 12:48:38.365162 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:48:38.365129    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96b6599b_adb3_4b7c_b3fa_df54c38b0f00.slice/crio-5db30a89a0e0871f23740449483b89c6c269019b2a0353a55fdc8b8ee3978ebc WatchSource:0}: Error finding container 5db30a89a0e0871f23740449483b89c6c269019b2a0353a55fdc8b8ee3978ebc: Status 404 returned error can't find the container with id 5db30a89a0e0871f23740449483b89c6c269019b2a0353a55fdc8b8ee3978ebc
Sep 16 12:48:38.592312 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:38.592277    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-77dd6" event={"ID":"463edcc3-887a-4637-9702-bc62f152f9ff","Type":"ContainerStarted","Data":"3c7ec4cc417acd8393a2b8c2b8d0e4425ad0cb419bae5f537fb4c62cb979c3fe"}
Sep 16 12:48:38.592491 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:38.592389    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-77dd6"
Sep 16 12:48:38.593865 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:38.593841    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-wwn5q" event={"ID":"96b6599b-adb3-4b7c-b3fa-df54c38b0f00","Type":"ContainerStarted","Data":"5db30a89a0e0871f23740449483b89c6c269019b2a0353a55fdc8b8ee3978ebc"}
Sep 16 12:48:38.617712 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:38.617631    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-77dd6" podStartSLOduration=2.260705666 podStartE2EDuration="4.617616982s" totalImagesPullingTime="2.356911316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:48:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:38.616189443 +0000 UTC m=+625.063422086" watchObservedRunningTime="2026-09-16 12:48:38.617616982 +0000 UTC m=+625.064849623"
Sep 16 12:48:40.606185 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:40.606154    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-wwn5q" event={"ID":"96b6599b-adb3-4b7c-b3fa-df54c38b0f00","Type":"ContainerStarted","Data":"cf93b83b31dfee5ec700d011edb3b44e57d13df06016879aff16037607e2b058"}
Sep 16 12:48:40.606617 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:40.606302    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-wwn5q"
Sep 16 12:48:40.627359 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:40.627311    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-wwn5q" podStartSLOduration=1.9141138519999998 podStartE2EDuration="3.627297325s" totalImagesPullingTime="1.713183473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:48:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:40.624697197 +0000 UTC m=+627.071929840" watchObservedRunningTime="2026-09-16 12:48:40.627297325 +0000 UTC m=+627.074529967"
Sep 16 12:48:44.525092 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:44.525058    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kvvrd"]
Sep 16 12:48:44.528488 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:44.528472    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kvvrd"
Sep 16 12:48:44.531680 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:44.531660    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-hsr4g\""
Sep 16 12:48:44.544555 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:44.544533    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kvvrd"]
Sep 16 12:48:44.617683 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:44.617657    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rknbl\" (UniqueName: \"kubernetes.io/projected/a0839f0c-e02f-4782-99d6-120be2e3365e-kube-api-access-rknbl\") pod \"kuadrant-operator-controller-manager-76f8f686bf-kvvrd\" (UID: \"a0839f0c-e02f-4782-99d6-120be2e3365e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kvvrd"
Sep 16 12:48:44.617810 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:44.617721    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0839f0c-e02f-4782-99d6-120be2e3365e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-kvvrd\" (UID: \"a0839f0c-e02f-4782-99d6-120be2e3365e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kvvrd"
Sep 16 12:48:44.718429 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:44.718402    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rknbl\" (UniqueName: \"kubernetes.io/projected/a0839f0c-e02f-4782-99d6-120be2e3365e-kube-api-access-rknbl\") pod \"kuadrant-operator-controller-manager-76f8f686bf-kvvrd\" (UID: \"a0839f0c-e02f-4782-99d6-120be2e3365e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kvvrd"
Sep 16 12:48:44.718586 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:44.718455    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0839f0c-e02f-4782-99d6-120be2e3365e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-kvvrd\" (UID: \"a0839f0c-e02f-4782-99d6-120be2e3365e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kvvrd"
Sep 16 12:48:44.718808 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:44.718791    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0839f0c-e02f-4782-99d6-120be2e3365e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-kvvrd\" (UID: \"a0839f0c-e02f-4782-99d6-120be2e3365e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kvvrd"
Sep 16 12:48:44.730794 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:44.730769    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rknbl\" (UniqueName: \"kubernetes.io/projected/a0839f0c-e02f-4782-99d6-120be2e3365e-kube-api-access-rknbl\") pod \"kuadrant-operator-controller-manager-76f8f686bf-kvvrd\" (UID: \"a0839f0c-e02f-4782-99d6-120be2e3365e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kvvrd"
Sep 16 12:48:44.838550 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:44.838484    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kvvrd"
Sep 16 12:48:45.121050 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:45.120960    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kvvrd"]
Sep 16 12:48:45.123443 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:48:45.123418    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0839f0c_e02f_4782_99d6_120be2e3365e.slice/crio-681d50234332481f79d990d5a1af3297dd21348afb95c8718ae01ec5183500b2 WatchSource:0}: Error finding container 681d50234332481f79d990d5a1af3297dd21348afb95c8718ae01ec5183500b2: Status 404 returned error can't find the container with id 681d50234332481f79d990d5a1af3297dd21348afb95c8718ae01ec5183500b2
Sep 16 12:48:45.631491 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:45.631461    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kvvrd" event={"ID":"a0839f0c-e02f-4782-99d6-120be2e3365e","Type":"ContainerStarted","Data":"681d50234332481f79d990d5a1af3297dd21348afb95c8718ae01ec5183500b2"}
Sep 16 12:48:49.603096 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:49.603003    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-77dd6"
Sep 16 12:48:49.659517 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:49.659483    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kvvrd" event={"ID":"a0839f0c-e02f-4782-99d6-120be2e3365e","Type":"ContainerStarted","Data":"1e0d0b766a4c4256560bbf4f14b7d05bc0ab6bc30bdd52cb8223bc53fa85e6bb"}
Sep 16 12:48:49.659724 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:49.659620    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kvvrd"
Sep 16 12:48:49.682157 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:49.682099    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kvvrd" podStartSLOduration=1.9472539979999999 podStartE2EDuration="5.682080599s" totalImagesPullingTime="3.734826601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:49.680507062 +0000 UTC m=+636.127739705" watchObservedRunningTime="2026-09-16 12:48:49.682080599 +0000 UTC m=+636.129313244"
Sep 16 12:48:51.613109 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:48:51.613081    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-wwn5q"
Sep 16 12:49:00.666123 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:00.666092    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kvvrd"
Sep 16 12:49:03.015490 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:03.015452    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kvvrd"]
Sep 16 12:49:03.016324 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:03.016290    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kvvrd" podUID="a0839f0c-e02f-4782-99d6-120be2e3365e" containerName="manager" containerID="cri-o://1e0d0b766a4c4256560bbf4f14b7d05bc0ab6bc30bdd52cb8223bc53fa85e6bb" gracePeriod=2
Sep 16 12:49:03.025080 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:03.025054    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kvvrd"]
Sep 16 12:49:03.041079 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:03.041051    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-smj76"]
Sep 16 12:49:03.041416 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:03.041402    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0839f0c-e02f-4782-99d6-120be2e3365e" containerName="manager"
Sep 16 12:49:03.041416 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:03.041417    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0839f0c-e02f-4782-99d6-120be2e3365e" containerName="manager"
Sep 16 12:49:03.041531 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:03.041497    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0839f0c-e02f-4782-99d6-120be2e3365e" containerName="manager"
Sep 16 12:49:03.044436 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:03.044402    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-smj76"
Sep 16 12:49:03.046995 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:49:03.046952    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-kvvrd\" is forbidden: User \"system:node:ip-10-0-133-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-190.ec2.internal' and this object" podUID="a0839f0c-e02f-4782-99d6-120be2e3365e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kvvrd"
Sep 16 12:49:03.069016 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:03.068986    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-smj76"]
Sep 16 12:49:03.075344 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:03.075311    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/27005eb2-835e-4b76-a9cb-f1ca3265b9e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-smj76\" (UID: \"27005eb2-835e-4b76-a9cb-f1ca3265b9e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-smj76"
Sep 16 12:49:03.075715 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:03.075690    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q9gv\" (UniqueName: \"kubernetes.io/projected/27005eb2-835e-4b76-a9cb-f1ca3265b9e8-kube-api-access-4q9gv\") pod \"kuadrant-operator-controller-manager-76f8f686bf-smj76\" (UID: \"27005eb2-835e-4b76-a9cb-f1ca3265b9e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-smj76"
Sep 16 12:49:03.176453 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:03.176420    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/27005eb2-835e-4b76-a9cb-f1ca3265b9e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-smj76\" (UID: \"27005eb2-835e-4b76-a9cb-f1ca3265b9e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-smj76"
Sep 16 12:49:03.176612 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:03.176485    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q9gv\" (UniqueName: \"kubernetes.io/projected/27005eb2-835e-4b76-a9cb-f1ca3265b9e8-kube-api-access-4q9gv\") pod \"kuadrant-operator-controller-manager-76f8f686bf-smj76\" (UID: \"27005eb2-835e-4b76-a9cb-f1ca3265b9e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-smj76"
Sep 16 12:49:03.176829 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:03.176809    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/27005eb2-835e-4b76-a9cb-f1ca3265b9e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-smj76\" (UID: \"27005eb2-835e-4b76-a9cb-f1ca3265b9e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-smj76"
Sep 16 12:49:03.185668 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:03.185640    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q9gv\" (UniqueName: \"kubernetes.io/projected/27005eb2-835e-4b76-a9cb-f1ca3265b9e8-kube-api-access-4q9gv\") pod \"kuadrant-operator-controller-manager-76f8f686bf-smj76\" (UID: \"27005eb2-835e-4b76-a9cb-f1ca3265b9e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-smj76"
Sep 16 12:49:03.265594 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:03.265534    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kvvrd"
Sep 16 12:49:03.267945 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:49:03.267921    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-kvvrd\" is forbidden: User \"system:node:ip-10-0-133-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-190.ec2.internal' and this object" podUID="a0839f0c-e02f-4782-99d6-120be2e3365e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kvvrd"
Sep 16 12:49:03.378443 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:03.378412    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rknbl\" (UniqueName: \"kubernetes.io/projected/a0839f0c-e02f-4782-99d6-120be2e3365e-kube-api-access-rknbl\") pod \"a0839f0c-e02f-4782-99d6-120be2e3365e\" (UID: \"a0839f0c-e02f-4782-99d6-120be2e3365e\") "
Sep 16 12:49:03.378606 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:03.378481    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0839f0c-e02f-4782-99d6-120be2e3365e-extensions-socket-volume\") pod \"a0839f0c-e02f-4782-99d6-120be2e3365e\" (UID: \"a0839f0c-e02f-4782-99d6-120be2e3365e\") "
Sep 16 12:49:03.378948 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:03.378921    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0839f0c-e02f-4782-99d6-120be2e3365e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a0839f0c-e02f-4782-99d6-120be2e3365e" (UID: "a0839f0c-e02f-4782-99d6-120be2e3365e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:49:03.380826 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:03.380807    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0839f0c-e02f-4782-99d6-120be2e3365e-kube-api-access-rknbl" (OuterVolumeSpecName: "kube-api-access-rknbl") pod "a0839f0c-e02f-4782-99d6-120be2e3365e" (UID: "a0839f0c-e02f-4782-99d6-120be2e3365e"). InnerVolumeSpecName "kube-api-access-rknbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:49:03.386968 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:03.386951    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-smj76"
Sep 16 12:49:03.480017 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:03.479983    2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0839f0c-e02f-4782-99d6-120be2e3365e-extensions-socket-volume\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:49:03.480017 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:03.480010    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rknbl\" (UniqueName: \"kubernetes.io/projected/a0839f0c-e02f-4782-99d6-120be2e3365e-kube-api-access-rknbl\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:49:03.664132 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:03.664097    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-smj76"]
Sep 16 12:49:03.666813 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:49:03.666788    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27005eb2_835e_4b76_a9cb_f1ca3265b9e8.slice/crio-1ec297acd415d0527e84291b9e5cf31d82f475628e97661a0801694f8821ce7a WatchSource:0}: Error finding container 1ec297acd415d0527e84291b9e5cf31d82f475628e97661a0801694f8821ce7a: Status 404 returned error can't find the container with id 1ec297acd415d0527e84291b9e5cf31d82f475628e97661a0801694f8821ce7a
Sep 16 12:49:03.730427 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:03.730397    2577 generic.go:356] "Generic (PLEG): container finished" podID="a0839f0c-e02f-4782-99d6-120be2e3365e" containerID="1e0d0b766a4c4256560bbf4f14b7d05bc0ab6bc30bdd52cb8223bc53fa85e6bb" exitCode=0
Sep 16 12:49:03.730545 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:03.730436    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kvvrd"
Sep 16 12:49:03.730545 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:03.730498    2577 scope.go:117] "RemoveContainer" containerID="1e0d0b766a4c4256560bbf4f14b7d05bc0ab6bc30bdd52cb8223bc53fa85e6bb"
Sep 16 12:49:03.732538 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:03.732510    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-smj76" event={"ID":"27005eb2-835e-4b76-a9cb-f1ca3265b9e8","Type":"ContainerStarted","Data":"1ec297acd415d0527e84291b9e5cf31d82f475628e97661a0801694f8821ce7a"}
Sep 16 12:49:03.732934 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:49:03.732909    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-kvvrd\" is forbidden: User \"system:node:ip-10-0-133-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-190.ec2.internal' and this object" podUID="a0839f0c-e02f-4782-99d6-120be2e3365e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kvvrd"
Sep 16 12:49:03.739709 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:03.739677    2577 scope.go:117] "RemoveContainer" containerID="1e0d0b766a4c4256560bbf4f14b7d05bc0ab6bc30bdd52cb8223bc53fa85e6bb"
Sep 16 12:49:03.739982 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:49:03.739963    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e0d0b766a4c4256560bbf4f14b7d05bc0ab6bc30bdd52cb8223bc53fa85e6bb\": container with ID starting with 1e0d0b766a4c4256560bbf4f14b7d05bc0ab6bc30bdd52cb8223bc53fa85e6bb not found: ID does not exist" containerID="1e0d0b766a4c4256560bbf4f14b7d05bc0ab6bc30bdd52cb8223bc53fa85e6bb"
Sep 16 12:49:03.740122 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:03.739990    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e0d0b766a4c4256560bbf4f14b7d05bc0ab6bc30bdd52cb8223bc53fa85e6bb"} err="failed to get container status \"1e0d0b766a4c4256560bbf4f14b7d05bc0ab6bc30bdd52cb8223bc53fa85e6bb\": rpc error: code = NotFound desc = could not find container \"1e0d0b766a4c4256560bbf4f14b7d05bc0ab6bc30bdd52cb8223bc53fa85e6bb\": container with ID starting with 1e0d0b766a4c4256560bbf4f14b7d05bc0ab6bc30bdd52cb8223bc53fa85e6bb not found: ID does not exist"
Sep 16 12:49:03.741627 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:49:03.741606    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-kvvrd\" is forbidden: User \"system:node:ip-10-0-133-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-190.ec2.internal' and this object" podUID="a0839f0c-e02f-4782-99d6-120be2e3365e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kvvrd"
Sep 16 12:49:04.052499 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:49:04.052462    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-kvvrd\" is forbidden: User \"system:node:ip-10-0-133-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-190.ec2.internal' and this object" podUID="a0839f0c-e02f-4782-99d6-120be2e3365e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kvvrd"
Sep 16 12:49:04.054117 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:04.054095    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0839f0c-e02f-4782-99d6-120be2e3365e" path="/var/lib/kubelet/pods/a0839f0c-e02f-4782-99d6-120be2e3365e/volumes"
Sep 16 12:49:04.740360 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:04.740325    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-smj76" event={"ID":"27005eb2-835e-4b76-a9cb-f1ca3265b9e8","Type":"ContainerStarted","Data":"3bd24faf23c3f8aa4b102c7368401351afc29768b6a7a558bbf5de976de8a42e"}
Sep 16 12:49:04.740551 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:04.740415    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-smj76"
Sep 16 12:49:04.763087 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:04.763016    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-smj76" podStartSLOduration=1.762997156 podStartE2EDuration="1.762997156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:04.761282572 +0000 UTC m=+651.208515208" watchObservedRunningTime="2026-09-16 12:49:04.762997156 +0000 UTC m=+651.210229798"
Sep 16 12:49:15.747295 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:15.747267    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-smj76"
Sep 16 12:49:19.595702 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:19.595662    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-smj76"]
Sep 16 12:49:19.596180 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:19.595899    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-smj76" podUID="27005eb2-835e-4b76-a9cb-f1ca3265b9e8" containerName="manager" containerID="cri-o://3bd24faf23c3f8aa4b102c7368401351afc29768b6a7a558bbf5de976de8a42e" gracePeriod=10
Sep 16 12:49:19.820247 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:19.820214    2577 generic.go:356] "Generic (PLEG): container finished" podID="27005eb2-835e-4b76-a9cb-f1ca3265b9e8" containerID="3bd24faf23c3f8aa4b102c7368401351afc29768b6a7a558bbf5de976de8a42e" exitCode=0
Sep 16 12:49:19.820389 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:19.820286    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-smj76" event={"ID":"27005eb2-835e-4b76-a9cb-f1ca3265b9e8","Type":"ContainerDied","Data":"3bd24faf23c3f8aa4b102c7368401351afc29768b6a7a558bbf5de976de8a42e"}
Sep 16 12:49:19.848275 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:19.848223    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-smj76"
Sep 16 12:49:19.888745 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:19.888715    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s9p8x"]
Sep 16 12:49:19.889300 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:19.889267    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27005eb2-835e-4b76-a9cb-f1ca3265b9e8" containerName="manager"
Sep 16 12:49:19.889300 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:19.889300    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="27005eb2-835e-4b76-a9cb-f1ca3265b9e8" containerName="manager"
Sep 16 12:49:19.889442 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:19.889425    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="27005eb2-835e-4b76-a9cb-f1ca3265b9e8" containerName="manager"
Sep 16 12:49:19.892576 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:19.892554    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s9p8x"
Sep 16 12:49:19.906868 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:19.906846    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s9p8x"]
Sep 16 12:49:19.918291 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:19.918271    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/27005eb2-835e-4b76-a9cb-f1ca3265b9e8-extensions-socket-volume\") pod \"27005eb2-835e-4b76-a9cb-f1ca3265b9e8\" (UID: \"27005eb2-835e-4b76-a9cb-f1ca3265b9e8\") "
Sep 16 12:49:19.918374 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:19.918336    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4q9gv\" (UniqueName: \"kubernetes.io/projected/27005eb2-835e-4b76-a9cb-f1ca3265b9e8-kube-api-access-4q9gv\") pod \"27005eb2-835e-4b76-a9cb-f1ca3265b9e8\" (UID: \"27005eb2-835e-4b76-a9cb-f1ca3265b9e8\") "
Sep 16 12:49:19.918598 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:19.918579    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27005eb2-835e-4b76-a9cb-f1ca3265b9e8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "27005eb2-835e-4b76-a9cb-f1ca3265b9e8" (UID: "27005eb2-835e-4b76-a9cb-f1ca3265b9e8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:49:19.920513 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:19.920486    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27005eb2-835e-4b76-a9cb-f1ca3265b9e8-kube-api-access-4q9gv" (OuterVolumeSpecName: "kube-api-access-4q9gv") pod "27005eb2-835e-4b76-a9cb-f1ca3265b9e8" (UID: "27005eb2-835e-4b76-a9cb-f1ca3265b9e8"). InnerVolumeSpecName "kube-api-access-4q9gv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:49:20.019115 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:20.019080    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wxdw\" (UniqueName: \"kubernetes.io/projected/70656ca4-0c05-44fd-8cdb-f29a8cf2d722-kube-api-access-2wxdw\") pod \"kuadrant-operator-controller-manager-5769f99d9f-s9p8x\" (UID: \"70656ca4-0c05-44fd-8cdb-f29a8cf2d722\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s9p8x"
Sep 16 12:49:20.019238 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:20.019189    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70656ca4-0c05-44fd-8cdb-f29a8cf2d722-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-s9p8x\" (UID: \"70656ca4-0c05-44fd-8cdb-f29a8cf2d722\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s9p8x"
Sep 16 12:49:20.019301 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:20.019238    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q9gv\" (UniqueName: \"kubernetes.io/projected/27005eb2-835e-4b76-a9cb-f1ca3265b9e8-kube-api-access-4q9gv\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:49:20.019301 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:20.019249    2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/27005eb2-835e-4b76-a9cb-f1ca3265b9e8-extensions-socket-volume\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:49:20.120401 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:20.120322    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70656ca4-0c05-44fd-8cdb-f29a8cf2d722-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-s9p8x\" (UID: \"70656ca4-0c05-44fd-8cdb-f29a8cf2d722\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s9p8x"
Sep 16 12:49:20.120536 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:20.120403    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wxdw\" (UniqueName: \"kubernetes.io/projected/70656ca4-0c05-44fd-8cdb-f29a8cf2d722-kube-api-access-2wxdw\") pod \"kuadrant-operator-controller-manager-5769f99d9f-s9p8x\" (UID: \"70656ca4-0c05-44fd-8cdb-f29a8cf2d722\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s9p8x"
Sep 16 12:49:20.120687 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:20.120667    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70656ca4-0c05-44fd-8cdb-f29a8cf2d722-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-s9p8x\" (UID: \"70656ca4-0c05-44fd-8cdb-f29a8cf2d722\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s9p8x"
Sep 16 12:49:20.134998 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:20.134969    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wxdw\" (UniqueName: \"kubernetes.io/projected/70656ca4-0c05-44fd-8cdb-f29a8cf2d722-kube-api-access-2wxdw\") pod \"kuadrant-operator-controller-manager-5769f99d9f-s9p8x\" (UID: \"70656ca4-0c05-44fd-8cdb-f29a8cf2d722\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s9p8x"
Sep 16 12:49:20.203158 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:20.203127    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s9p8x"
Sep 16 12:49:20.489715 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:20.489628    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s9p8x"]
Sep 16 12:49:20.491945 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:49:20.491919    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70656ca4_0c05_44fd_8cdb_f29a8cf2d722.slice/crio-3119c7dbfa133a22689704dcca82ec0c2bc44d1bb6e84660af7e8690d7115c83 WatchSource:0}: Error finding container 3119c7dbfa133a22689704dcca82ec0c2bc44d1bb6e84660af7e8690d7115c83: Status 404 returned error can't find the container with id 3119c7dbfa133a22689704dcca82ec0c2bc44d1bb6e84660af7e8690d7115c83
Sep 16 12:49:20.826921 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:20.826892    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-smj76"
Sep 16 12:49:20.827375 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:20.826886    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-smj76" event={"ID":"27005eb2-835e-4b76-a9cb-f1ca3265b9e8","Type":"ContainerDied","Data":"1ec297acd415d0527e84291b9e5cf31d82f475628e97661a0801694f8821ce7a"}
Sep 16 12:49:20.827375 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:20.827018    2577 scope.go:117] "RemoveContainer" containerID="3bd24faf23c3f8aa4b102c7368401351afc29768b6a7a558bbf5de976de8a42e"
Sep 16 12:49:20.829089 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:20.829063    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s9p8x" event={"ID":"70656ca4-0c05-44fd-8cdb-f29a8cf2d722","Type":"ContainerStarted","Data":"3a9902c30c0b87174efab44511d404e2cc33184f6f2b6ee3c6daa72e37d9244d"}
Sep 16 12:49:20.829221 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:20.829120    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s9p8x"
Sep 16 12:49:20.829221 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:20.829147    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s9p8x" event={"ID":"70656ca4-0c05-44fd-8cdb-f29a8cf2d722","Type":"ContainerStarted","Data":"3119c7dbfa133a22689704dcca82ec0c2bc44d1bb6e84660af7e8690d7115c83"}
Sep 16 12:49:20.856997 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:20.856951    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s9p8x" podStartSLOduration=1.856937024 podStartE2EDuration="1.856937024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:20.855762888 +0000 UTC m=+667.302995530" watchObservedRunningTime="2026-09-16 12:49:20.856937024 +0000 UTC m=+667.304169666"
Sep 16 12:49:20.882094 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:20.882072    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-smj76"]
Sep 16 12:49:20.892480 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:20.892451    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-smj76"]
Sep 16 12:49:22.054650 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:22.054619    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27005eb2-835e-4b76-a9cb-f1ca3265b9e8" path="/var/lib/kubelet/pods/27005eb2-835e-4b76-a9cb-f1ca3265b9e8/volumes"
Sep 16 12:49:31.838126 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:31.838096    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s9p8x"
Sep 16 12:49:36.106718 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:36.106688    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"]
Sep 16 12:49:36.111420 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:36.111396    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"
Sep 16 12:49:36.115173 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:36.115148    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4cx2s\""
Sep 16 12:49:36.131968 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:36.131945    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"]
Sep 16 12:49:36.268522 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:36.268494    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be45e9ae-2a43-4cd9-bd86-541bc2a97291-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9rfz6\" (UID: \"be45e9ae-2a43-4cd9-bd86-541bc2a97291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"
Sep 16 12:49:36.268686 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:36.268536    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be45e9ae-2a43-4cd9-bd86-541bc2a97291-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9rfz6\" (UID: \"be45e9ae-2a43-4cd9-bd86-541bc2a97291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"
Sep 16 12:49:36.268686 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:36.268570    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be45e9ae-2a43-4cd9-bd86-541bc2a97291-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9rfz6\" (UID: \"be45e9ae-2a43-4cd9-bd86-541bc2a97291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"
Sep 16 12:49:36.268686 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:36.268586    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be45e9ae-2a43-4cd9-bd86-541bc2a97291-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9rfz6\" (UID: \"be45e9ae-2a43-4cd9-bd86-541bc2a97291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"
Sep 16 12:49:36.268686 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:36.268629    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be45e9ae-2a43-4cd9-bd86-541bc2a97291-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9rfz6\" (UID: \"be45e9ae-2a43-4cd9-bd86-541bc2a97291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"
Sep 16 12:49:36.268686 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:36.268660    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be45e9ae-2a43-4cd9-bd86-541bc2a97291-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9rfz6\" (UID: \"be45e9ae-2a43-4cd9-bd86-541bc2a97291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"
Sep 16 12:49:36.268686 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:36.268677    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mfpj\" (UniqueName: \"kubernetes.io/projected/be45e9ae-2a43-4cd9-bd86-541bc2a97291-kube-api-access-7mfpj\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9rfz6\" (UID: \"be45e9ae-2a43-4cd9-bd86-541bc2a97291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"
Sep 16 12:49:36.268886 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:36.268722    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be45e9ae-2a43-4cd9-bd86-541bc2a97291-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9rfz6\" (UID: \"be45e9ae-2a43-4cd9-bd86-541bc2a97291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"
Sep 16 12:49:36.268886 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:36.268758    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be45e9ae-2a43-4cd9-bd86-541bc2a97291-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9rfz6\" (UID: \"be45e9ae-2a43-4cd9-bd86-541bc2a97291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"
Sep 16 12:49:36.369460 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:36.369370    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be45e9ae-2a43-4cd9-bd86-541bc2a97291-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9rfz6\" (UID: \"be45e9ae-2a43-4cd9-bd86-541bc2a97291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"
Sep 16 12:49:36.369460 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:36.369426    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be45e9ae-2a43-4cd9-bd86-541bc2a97291-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9rfz6\" (UID: \"be45e9ae-2a43-4cd9-bd86-541bc2a97291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"
Sep 16 12:49:36.369460 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:36.369462    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be45e9ae-2a43-4cd9-bd86-541bc2a97291-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9rfz6\" (UID: \"be45e9ae-2a43-4cd9-bd86-541bc2a97291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"
Sep 16 12:49:36.369745 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:36.369497    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be45e9ae-2a43-4cd9-bd86-541bc2a97291-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9rfz6\" (UID: \"be45e9ae-2a43-4cd9-bd86-541bc2a97291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"
Sep 16 12:49:36.369745 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:36.369519    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be45e9ae-2a43-4cd9-bd86-541bc2a97291-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9rfz6\" (UID: \"be45e9ae-2a43-4cd9-bd86-541bc2a97291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"
Sep 16 12:49:36.369745 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:36.369548    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be45e9ae-2a43-4cd9-bd86-541bc2a97291-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9rfz6\" (UID: \"be45e9ae-2a43-4cd9-bd86-541bc2a97291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"
Sep 16 12:49:36.369745 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:36.369588    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be45e9ae-2a43-4cd9-bd86-541bc2a97291-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9rfz6\" (UID: \"be45e9ae-2a43-4cd9-bd86-541bc2a97291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"
Sep 16 12:49:36.369745 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:36.369616    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mfpj\" (UniqueName: \"kubernetes.io/projected/be45e9ae-2a43-4cd9-bd86-541bc2a97291-kube-api-access-7mfpj\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9rfz6\" (UID: \"be45e9ae-2a43-4cd9-bd86-541bc2a97291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"
Sep 16 12:49:36.369745 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:36.369649    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be45e9ae-2a43-4cd9-bd86-541bc2a97291-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9rfz6\" (UID: \"be45e9ae-2a43-4cd9-bd86-541bc2a97291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"
Sep 16 12:49:36.370078 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:36.369953    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be45e9ae-2a43-4cd9-bd86-541bc2a97291-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9rfz6\" (UID: \"be45e9ae-2a43-4cd9-bd86-541bc2a97291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"
Sep 16 12:49:36.370144 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:36.370087    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be45e9ae-2a43-4cd9-bd86-541bc2a97291-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9rfz6\" (UID: \"be45e9ae-2a43-4cd9-bd86-541bc2a97291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"
Sep 16 12:49:36.370144 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:36.370084    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be45e9ae-2a43-4cd9-bd86-541bc2a97291-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9rfz6\" (UID: \"be45e9ae-2a43-4cd9-bd86-541bc2a97291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"
Sep 16 12:49:36.370253 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:36.370151    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be45e9ae-2a43-4cd9-bd86-541bc2a97291-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9rfz6\" (UID: \"be45e9ae-2a43-4cd9-bd86-541bc2a97291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"
Sep 16 12:49:36.370253 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:36.370231    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be45e9ae-2a43-4cd9-bd86-541bc2a97291-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9rfz6\" (UID: \"be45e9ae-2a43-4cd9-bd86-541bc2a97291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"
Sep 16 12:49:36.371876 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:36.371852    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be45e9ae-2a43-4cd9-bd86-541bc2a97291-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9rfz6\" (UID: \"be45e9ae-2a43-4cd9-bd86-541bc2a97291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"
Sep 16 12:49:36.372364 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:36.372343    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be45e9ae-2a43-4cd9-bd86-541bc2a97291-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9rfz6\" (UID: \"be45e9ae-2a43-4cd9-bd86-541bc2a97291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"
Sep 16 12:49:36.383764 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:36.383742    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be45e9ae-2a43-4cd9-bd86-541bc2a97291-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9rfz6\" (UID: \"be45e9ae-2a43-4cd9-bd86-541bc2a97291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"
Sep 16 12:49:36.383889 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:36.383873    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mfpj\" (UniqueName: \"kubernetes.io/projected/be45e9ae-2a43-4cd9-bd86-541bc2a97291-kube-api-access-7mfpj\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9rfz6\" (UID: \"be45e9ae-2a43-4cd9-bd86-541bc2a97291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"
Sep 16 12:49:36.421840 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:36.421818    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"
Sep 16 12:49:36.703062 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:36.703020    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"]
Sep 16 12:49:36.704979 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:49:36.704941    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe45e9ae_2a43_4cd9_bd86_541bc2a97291.slice/crio-86bb8b295763fb3843cf67eb56284574f86478bdaf444d6c72140e1e0e5ed101 WatchSource:0}: Error finding container 86bb8b295763fb3843cf67eb56284574f86478bdaf444d6c72140e1e0e5ed101: Status 404 returned error can't find the container with id 86bb8b295763fb3843cf67eb56284574f86478bdaf444d6c72140e1e0e5ed101
Sep 16 12:49:36.707127 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:36.707096    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:49:36.707204 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:36.707157    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:49:36.707204 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:36.707186    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:49:36.919493 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:36.919455    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6" event={"ID":"be45e9ae-2a43-4cd9-bd86-541bc2a97291","Type":"ContainerStarted","Data":"e909ec826814cd1f95f90d890350b1f50f7f132ccc2ba33294d7f5320094aad1"}
Sep 16 12:49:36.919673 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:36.919561    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6" event={"ID":"be45e9ae-2a43-4cd9-bd86-541bc2a97291","Type":"ContainerStarted","Data":"86bb8b295763fb3843cf67eb56284574f86478bdaf444d6c72140e1e0e5ed101"}
Sep 16 12:49:36.947134 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:36.947088    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6" podStartSLOduration=0.947069629 podStartE2EDuration="947.069629ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:36.945974845 +0000 UTC m=+683.393207510" watchObservedRunningTime="2026-09-16 12:49:36.947069629 +0000 UTC m=+683.394302341"
Sep 16 12:49:37.422137 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:37.422104    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"
Sep 16 12:49:37.427347 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:37.427326    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"
Sep 16 12:49:37.925807 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:37.925775    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"
Sep 16 12:49:37.926640 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:37.926619    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"
Sep 16 12:49:53.566563 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:53.566529    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qnb4s"]
Sep 16 12:49:53.570122 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:53.570101    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-qnb4s"
Sep 16 12:49:53.573208 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:53.573189    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9hx2j\""
Sep 16 12:49:53.580447 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:53.580414    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qnb4s"]
Sep 16 12:49:53.621795 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:53.621768    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dmb5\" (UniqueName: \"kubernetes.io/projected/6b10fd32-b5b3-41f4-af15-4741ebb87fcf-kube-api-access-2dmb5\") pod \"authorino-7bcfd8f9cd-qnb4s\" (UID: \"6b10fd32-b5b3-41f4-af15-4741ebb87fcf\") " pod="kuadrant-system/authorino-7bcfd8f9cd-qnb4s"
Sep 16 12:49:53.722680 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:53.722640    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dmb5\" (UniqueName: \"kubernetes.io/projected/6b10fd32-b5b3-41f4-af15-4741ebb87fcf-kube-api-access-2dmb5\") pod \"authorino-7bcfd8f9cd-qnb4s\" (UID: \"6b10fd32-b5b3-41f4-af15-4741ebb87fcf\") " pod="kuadrant-system/authorino-7bcfd8f9cd-qnb4s"
Sep 16 12:49:53.731978 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:53.731950    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dmb5\" (UniqueName: \"kubernetes.io/projected/6b10fd32-b5b3-41f4-af15-4741ebb87fcf-kube-api-access-2dmb5\") pod \"authorino-7bcfd8f9cd-qnb4s\" (UID: \"6b10fd32-b5b3-41f4-af15-4741ebb87fcf\") " pod="kuadrant-system/authorino-7bcfd8f9cd-qnb4s"
Sep 16 12:49:53.881546 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:53.881464    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-qnb4s"
Sep 16 12:49:54.153081 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:54.153058    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qnb4s"]
Sep 16 12:49:54.155169 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:49:54.155140    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b10fd32_b5b3_41f4_af15_4741ebb87fcf.slice/crio-4707ed2e5f94c35f3155c6511d345dfb7fc3b7c8c27128d10e47f0b7418552ef WatchSource:0}: Error finding container 4707ed2e5f94c35f3155c6511d345dfb7fc3b7c8c27128d10e47f0b7418552ef: Status 404 returned error can't find the container with id 4707ed2e5f94c35f3155c6511d345dfb7fc3b7c8c27128d10e47f0b7418552ef
Sep 16 12:49:55.012749 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:55.012706    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-qnb4s" event={"ID":"6b10fd32-b5b3-41f4-af15-4741ebb87fcf","Type":"ContainerStarted","Data":"4707ed2e5f94c35f3155c6511d345dfb7fc3b7c8c27128d10e47f0b7418552ef"}
Sep 16 12:49:56.019382 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:56.019300    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-qnb4s" event={"ID":"6b10fd32-b5b3-41f4-af15-4741ebb87fcf","Type":"ContainerStarted","Data":"0025e1e4e47e7c045a83da64e6b2beb803330b715127bd640a6bd08048739b5d"}
Sep 16 12:49:56.040676 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:56.040628    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-qnb4s" podStartSLOduration=1.432426962 podStartE2EDuration="3.040613005s" totalImagesPullingTime="1.608186043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:56.038187826 +0000 UTC m=+702.485420465" watchObservedRunningTime="2026-09-16 12:49:56.040613005 +0000 UTC m=+702.487845647"
Sep 16 12:49:56.429502 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:56.429430    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qnb4s"]
Sep 16 12:49:58.028977 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:58.028940    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-qnb4s" podUID="6b10fd32-b5b3-41f4-af15-4741ebb87fcf" containerName="authorino" containerID="cri-o://0025e1e4e47e7c045a83da64e6b2beb803330b715127bd640a6bd08048739b5d" gracePeriod=30
Sep 16 12:49:58.285127 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:58.285071    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-qnb4s"
Sep 16 12:49:58.365603 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:58.365575    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dmb5\" (UniqueName: \"kubernetes.io/projected/6b10fd32-b5b3-41f4-af15-4741ebb87fcf-kube-api-access-2dmb5\") pod \"6b10fd32-b5b3-41f4-af15-4741ebb87fcf\" (UID: \"6b10fd32-b5b3-41f4-af15-4741ebb87fcf\") "
Sep 16 12:49:58.367799 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:58.367767    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b10fd32-b5b3-41f4-af15-4741ebb87fcf-kube-api-access-2dmb5" (OuterVolumeSpecName: "kube-api-access-2dmb5") pod "6b10fd32-b5b3-41f4-af15-4741ebb87fcf" (UID: "6b10fd32-b5b3-41f4-af15-4741ebb87fcf"). InnerVolumeSpecName "kube-api-access-2dmb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:49:58.466880 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:58.466849    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dmb5\" (UniqueName: \"kubernetes.io/projected/6b10fd32-b5b3-41f4-af15-4741ebb87fcf-kube-api-access-2dmb5\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:49:59.034492 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:59.034453    2577 generic.go:356] "Generic (PLEG): container finished" podID="6b10fd32-b5b3-41f4-af15-4741ebb87fcf" containerID="0025e1e4e47e7c045a83da64e6b2beb803330b715127bd640a6bd08048739b5d" exitCode=0
Sep 16 12:49:59.034928 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:59.034501    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-qnb4s"
Sep 16 12:49:59.034928 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:59.034491    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-qnb4s" event={"ID":"6b10fd32-b5b3-41f4-af15-4741ebb87fcf","Type":"ContainerDied","Data":"0025e1e4e47e7c045a83da64e6b2beb803330b715127bd640a6bd08048739b5d"}
Sep 16 12:49:59.034928 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:59.034615    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-qnb4s" event={"ID":"6b10fd32-b5b3-41f4-af15-4741ebb87fcf","Type":"ContainerDied","Data":"4707ed2e5f94c35f3155c6511d345dfb7fc3b7c8c27128d10e47f0b7418552ef"}
Sep 16 12:49:59.034928 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:59.034633    2577 scope.go:117] "RemoveContainer" containerID="0025e1e4e47e7c045a83da64e6b2beb803330b715127bd640a6bd08048739b5d"
Sep 16 12:49:59.045988 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:59.045971    2577 scope.go:117] "RemoveContainer" containerID="0025e1e4e47e7c045a83da64e6b2beb803330b715127bd640a6bd08048739b5d"
Sep 16 12:49:59.046262 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:49:59.046237    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0025e1e4e47e7c045a83da64e6b2beb803330b715127bd640a6bd08048739b5d\": container with ID starting with 0025e1e4e47e7c045a83da64e6b2beb803330b715127bd640a6bd08048739b5d not found: ID does not exist" containerID="0025e1e4e47e7c045a83da64e6b2beb803330b715127bd640a6bd08048739b5d"
Sep 16 12:49:59.046322 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:59.046270    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0025e1e4e47e7c045a83da64e6b2beb803330b715127bd640a6bd08048739b5d"} err="failed to get container status \"0025e1e4e47e7c045a83da64e6b2beb803330b715127bd640a6bd08048739b5d\": rpc error: code = NotFound desc = could not find container \"0025e1e4e47e7c045a83da64e6b2beb803330b715127bd640a6bd08048739b5d\": container with ID starting with 0025e1e4e47e7c045a83da64e6b2beb803330b715127bd640a6bd08048739b5d not found: ID does not exist"
Sep 16 12:49:59.058175 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:59.058147    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qnb4s"]
Sep 16 12:49:59.061364 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:49:59.061330    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qnb4s"]
Sep 16 12:50:00.054598 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:00.054564    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b10fd32-b5b3-41f4-af15-4741ebb87fcf" path="/var/lib/kubelet/pods/6b10fd32-b5b3-41f4-af15-4741ebb87fcf/volumes"
Sep 16 12:50:22.494677 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:22.494645    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-zb4lw"]
Sep 16 12:50:22.502925 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:22.502892    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b10fd32-b5b3-41f4-af15-4741ebb87fcf" containerName="authorino"
Sep 16 12:50:22.502925 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:22.502923    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b10fd32-b5b3-41f4-af15-4741ebb87fcf" containerName="authorino"
Sep 16 12:50:22.503139 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:22.503060    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b10fd32-b5b3-41f4-af15-4741ebb87fcf" containerName="authorino"
Sep 16 12:50:22.510016 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:22.509932    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-zb4lw"
Sep 16 12:50:22.513045 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:22.513009    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9hx2j\""
Sep 16 12:50:22.513161 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:22.513126    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-zb4lw"]
Sep 16 12:50:22.613086 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:22.613043    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-zb4lw"]
Sep 16 12:50:22.613382 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:50:22.613361    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-5755t], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-zb4lw" podUID="877e82c3-7920-490e-954f-face70111190"
Sep 16 12:50:22.650128 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:22.650094    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-pqxgb"]
Sep 16 12:50:22.657951 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:22.657915    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-pqxgb"
Sep 16 12:50:22.669094 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:22.667193    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-pqxgb"]
Sep 16 12:50:22.669094 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:22.667406    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 16 12:50:22.675595 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:22.675566    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5755t\" (UniqueName: \"kubernetes.io/projected/877e82c3-7920-490e-954f-face70111190-kube-api-access-5755t\") pod \"authorino-557f876f69-zb4lw\" (UID: \"877e82c3-7920-490e-954f-face70111190\") " pod="kuadrant-system/authorino-557f876f69-zb4lw"
Sep 16 12:50:22.776704 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:22.776657    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk2zm\" (UniqueName: \"kubernetes.io/projected/f1fa8c07-5416-45cd-98e0-962141fd82c6-kube-api-access-dk2zm\") pod \"authorino-6d4dfc649f-pqxgb\" (UID: \"f1fa8c07-5416-45cd-98e0-962141fd82c6\") " pod="kuadrant-system/authorino-6d4dfc649f-pqxgb"
Sep 16 12:50:22.776903 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:22.776713    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5755t\" (UniqueName: \"kubernetes.io/projected/877e82c3-7920-490e-954f-face70111190-kube-api-access-5755t\") pod \"authorino-557f876f69-zb4lw\" (UID: \"877e82c3-7920-490e-954f-face70111190\") " pod="kuadrant-system/authorino-557f876f69-zb4lw"
Sep 16 12:50:22.776903 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:22.776807    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f1fa8c07-5416-45cd-98e0-962141fd82c6-tls-cert\") pod \"authorino-6d4dfc649f-pqxgb\" (UID: \"f1fa8c07-5416-45cd-98e0-962141fd82c6\") " pod="kuadrant-system/authorino-6d4dfc649f-pqxgb"
Sep 16 12:50:22.778169 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:22.778143    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-pqxgb"]
Sep 16 12:50:22.778418 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:50:22.778393    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-dk2zm tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-pqxgb" podUID="f1fa8c07-5416-45cd-98e0-962141fd82c6"
Sep 16 12:50:22.789650 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:22.789622    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5755t\" (UniqueName: \"kubernetes.io/projected/877e82c3-7920-490e-954f-face70111190-kube-api-access-5755t\") pod \"authorino-557f876f69-zb4lw\" (UID: \"877e82c3-7920-490e-954f-face70111190\") " pod="kuadrant-system/authorino-557f876f69-zb4lw"
Sep 16 12:50:22.823316 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:22.823289    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5fcdd846cb-f5q2s"]
Sep 16 12:50:22.827914 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:22.827897    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fcdd846cb-f5q2s"
Sep 16 12:50:22.837093 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:22.837071    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5fcdd846cb-f5q2s"]
Sep 16 12:50:22.877432 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:22.877395    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk2zm\" (UniqueName: \"kubernetes.io/projected/f1fa8c07-5416-45cd-98e0-962141fd82c6-kube-api-access-dk2zm\") pod \"authorino-6d4dfc649f-pqxgb\" (UID: \"f1fa8c07-5416-45cd-98e0-962141fd82c6\") " pod="kuadrant-system/authorino-6d4dfc649f-pqxgb"
Sep 16 12:50:22.877561 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:22.877454    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f1fa8c07-5416-45cd-98e0-962141fd82c6-tls-cert\") pod \"authorino-6d4dfc649f-pqxgb\" (UID: \"f1fa8c07-5416-45cd-98e0-962141fd82c6\") " pod="kuadrant-system/authorino-6d4dfc649f-pqxgb"
Sep 16 12:50:22.879909 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:22.879882    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f1fa8c07-5416-45cd-98e0-962141fd82c6-tls-cert\") pod \"authorino-6d4dfc649f-pqxgb\" (UID: \"f1fa8c07-5416-45cd-98e0-962141fd82c6\") " pod="kuadrant-system/authorino-6d4dfc649f-pqxgb"
Sep 16 12:50:22.886642 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:22.886621    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk2zm\" (UniqueName: \"kubernetes.io/projected/f1fa8c07-5416-45cd-98e0-962141fd82c6-kube-api-access-dk2zm\") pod \"authorino-6d4dfc649f-pqxgb\" (UID: \"f1fa8c07-5416-45cd-98e0-962141fd82c6\") " pod="kuadrant-system/authorino-6d4dfc649f-pqxgb"
Sep 16 12:50:22.978624 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:22.978584    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-294mt\" (UniqueName: \"kubernetes.io/projected/299d7f29-6a45-479a-a225-04de47548ab1-kube-api-access-294mt\") pod \"authorino-5fcdd846cb-f5q2s\" (UID: \"299d7f29-6a45-479a-a225-04de47548ab1\") " pod="kuadrant-system/authorino-5fcdd846cb-f5q2s"
Sep 16 12:50:22.978624 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:22.978627    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/299d7f29-6a45-479a-a225-04de47548ab1-tls-cert\") pod \"authorino-5fcdd846cb-f5q2s\" (UID: \"299d7f29-6a45-479a-a225-04de47548ab1\") " pod="kuadrant-system/authorino-5fcdd846cb-f5q2s"
Sep 16 12:50:23.080144 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:23.080047    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-294mt\" (UniqueName: \"kubernetes.io/projected/299d7f29-6a45-479a-a225-04de47548ab1-kube-api-access-294mt\") pod \"authorino-5fcdd846cb-f5q2s\" (UID: \"299d7f29-6a45-479a-a225-04de47548ab1\") " pod="kuadrant-system/authorino-5fcdd846cb-f5q2s"
Sep 16 12:50:23.080144 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:23.080100    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/299d7f29-6a45-479a-a225-04de47548ab1-tls-cert\") pod \"authorino-5fcdd846cb-f5q2s\" (UID: \"299d7f29-6a45-479a-a225-04de47548ab1\") " pod="kuadrant-system/authorino-5fcdd846cb-f5q2s"
Sep 16 12:50:23.082680 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:23.082655    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/299d7f29-6a45-479a-a225-04de47548ab1-tls-cert\") pod \"authorino-5fcdd846cb-f5q2s\" (UID: \"299d7f29-6a45-479a-a225-04de47548ab1\") " pod="kuadrant-system/authorino-5fcdd846cb-f5q2s"
Sep 16 12:50:23.090516 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:23.090492    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-294mt\" (UniqueName: \"kubernetes.io/projected/299d7f29-6a45-479a-a225-04de47548ab1-kube-api-access-294mt\") pod \"authorino-5fcdd846cb-f5q2s\" (UID: \"299d7f29-6a45-479a-a225-04de47548ab1\") " pod="kuadrant-system/authorino-5fcdd846cb-f5q2s"
Sep 16 12:50:23.137670 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:23.137648    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fcdd846cb-f5q2s"
Sep 16 12:50:23.148106 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:23.148086    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-pqxgb"
Sep 16 12:50:23.148289 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:23.148269    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-zb4lw"
Sep 16 12:50:23.156730 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:23.156707    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-pqxgb"
Sep 16 12:50:23.159797 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:23.159779    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-zb4lw"
Sep 16 12:50:23.281540 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:23.281512    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f1fa8c07-5416-45cd-98e0-962141fd82c6-tls-cert\") pod \"f1fa8c07-5416-45cd-98e0-962141fd82c6\" (UID: \"f1fa8c07-5416-45cd-98e0-962141fd82c6\") "
Sep 16 12:50:23.281716 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:23.281565    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dk2zm\" (UniqueName: \"kubernetes.io/projected/f1fa8c07-5416-45cd-98e0-962141fd82c6-kube-api-access-dk2zm\") pod \"f1fa8c07-5416-45cd-98e0-962141fd82c6\" (UID: \"f1fa8c07-5416-45cd-98e0-962141fd82c6\") "
Sep 16 12:50:23.281716 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:23.281613    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5755t\" (UniqueName: \"kubernetes.io/projected/877e82c3-7920-490e-954f-face70111190-kube-api-access-5755t\") pod \"877e82c3-7920-490e-954f-face70111190\" (UID: \"877e82c3-7920-490e-954f-face70111190\") "
Sep 16 12:50:23.283881 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:23.283846    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1fa8c07-5416-45cd-98e0-962141fd82c6-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "f1fa8c07-5416-45cd-98e0-962141fd82c6" (UID: "f1fa8c07-5416-45cd-98e0-962141fd82c6"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:50:23.283881 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:23.283862    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1fa8c07-5416-45cd-98e0-962141fd82c6-kube-api-access-dk2zm" (OuterVolumeSpecName: "kube-api-access-dk2zm") pod "f1fa8c07-5416-45cd-98e0-962141fd82c6" (UID: "f1fa8c07-5416-45cd-98e0-962141fd82c6"). InnerVolumeSpecName "kube-api-access-dk2zm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:50:23.283881 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:23.283849    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/877e82c3-7920-490e-954f-face70111190-kube-api-access-5755t" (OuterVolumeSpecName: "kube-api-access-5755t") pod "877e82c3-7920-490e-954f-face70111190" (UID: "877e82c3-7920-490e-954f-face70111190"). InnerVolumeSpecName "kube-api-access-5755t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:50:23.382938 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:23.382853    2577 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f1fa8c07-5416-45cd-98e0-962141fd82c6-tls-cert\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:50:23.382938 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:23.382885    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk2zm\" (UniqueName: \"kubernetes.io/projected/f1fa8c07-5416-45cd-98e0-962141fd82c6-kube-api-access-dk2zm\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:50:23.382938 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:23.382899    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5755t\" (UniqueName: \"kubernetes.io/projected/877e82c3-7920-490e-954f-face70111190-kube-api-access-5755t\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:50:23.410949 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:23.410928    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5fcdd846cb-f5q2s"]
Sep 16 12:50:23.412994 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:50:23.412968    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod299d7f29_6a45_479a_a225_04de47548ab1.slice/crio-371e61c33e78e72b9a0dc1cdae373422ca31564aa73b9ae9688f2d3eae7f0a08 WatchSource:0}: Error finding container 371e61c33e78e72b9a0dc1cdae373422ca31564aa73b9ae9688f2d3eae7f0a08: Status 404 returned error can't find the container with id 371e61c33e78e72b9a0dc1cdae373422ca31564aa73b9ae9688f2d3eae7f0a08
Sep 16 12:50:24.153830 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:24.153797    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fcdd846cb-f5q2s" event={"ID":"299d7f29-6a45-479a-a225-04de47548ab1","Type":"ContainerStarted","Data":"371e61c33e78e72b9a0dc1cdae373422ca31564aa73b9ae9688f2d3eae7f0a08"}
Sep 16 12:50:24.153830 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:24.153816    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-zb4lw"
Sep 16 12:50:24.153830 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:24.153830    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-pqxgb"
Sep 16 12:50:24.183630 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:24.183600    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-zb4lw"]
Sep 16 12:50:24.187238 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:24.187215    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-zb4lw"]
Sep 16 12:50:24.225636 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:24.225596    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-pqxgb"]
Sep 16 12:50:24.228418 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:24.228391    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-pqxgb"]
Sep 16 12:50:25.159937 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:25.159900    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fcdd846cb-f5q2s" event={"ID":"299d7f29-6a45-479a-a225-04de47548ab1","Type":"ContainerStarted","Data":"1a84b8c29ccf7b1f239286bf9eec1685060fc7098cda08db7b4f48d2f60e38e9"}
Sep 16 12:50:25.178335 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:25.178283    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5fcdd846cb-f5q2s" podStartSLOduration=2.290617061 podStartE2EDuration="3.17826727s" totalImagesPullingTime="887.650209ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:50:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:50:25.177324712 +0000 UTC m=+731.624557355" watchObservedRunningTime="2026-09-16 12:50:25.17826727 +0000 UTC m=+731.625499912"
Sep 16 12:50:26.055844 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:26.055810    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="877e82c3-7920-490e-954f-face70111190" path="/var/lib/kubelet/pods/877e82c3-7920-490e-954f-face70111190/volumes"
Sep 16 12:50:26.056087 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:50:26.056071    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1fa8c07-5416-45cd-98e0-962141fd82c6" path="/var/lib/kubelet/pods/f1fa8c07-5416-45cd-98e0-962141fd82c6/volumes"
Sep 16 12:51:18.422535 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:51:18.422496    2577 generic.go:356] "Generic (PLEG): container finished" podID="be45e9ae-2a43-4cd9-bd86-541bc2a97291" containerID="e909ec826814cd1f95f90d890350b1f50f7f132ccc2ba33294d7f5320094aad1" exitCode=0
Sep 16 12:51:18.423249 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:51:18.422568    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6" event={"ID":"be45e9ae-2a43-4cd9-bd86-541bc2a97291","Type":"ContainerDied","Data":"e909ec826814cd1f95f90d890350b1f50f7f132ccc2ba33294d7f5320094aad1"}
Sep 16 12:51:18.423249 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:51:18.422944    2577 scope.go:117] "RemoveContainer" containerID="e909ec826814cd1f95f90d890350b1f50f7f132ccc2ba33294d7f5320094aad1"
Sep 16 12:51:18.423993 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:51:18.423966    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:51:18.424068 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:51:18.424052    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:51:18.424114 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:51:18.424084    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:51:19.430044 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:51:19.429994    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6" event={"ID":"be45e9ae-2a43-4cd9-bd86-541bc2a97291","Type":"ContainerStarted","Data":"a2c0534d34cd07a05f6f718a7b62fea2b9b7a1da80c1c1f222d17a5fac5a2402"}
Sep 16 12:51:20.422458 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:51:20.422417    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"
Sep 16 12:51:20.423627 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:51:20.423598    2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-9rfz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Sep 16 12:51:20.423732 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:51:20.423650    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6" podUID="be45e9ae-2a43-4cd9-bd86-541bc2a97291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 16 12:51:21.422394 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:51:21.422357    2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-9rfz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Sep 16 12:51:21.422773 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:51:21.422421    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6" podUID="be45e9ae-2a43-4cd9-bd86-541bc2a97291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 16 12:51:22.423110 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:51:22.423069    2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-9rfz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Sep 16 12:51:22.423513 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:51:22.423130    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6" podUID="be45e9ae-2a43-4cd9-bd86-541bc2a97291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 16 12:51:22.925816 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:51:22.925780    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"
Sep 16 12:51:23.423207 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:51:23.423170    2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-9rfz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Sep 16 12:51:23.423653 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:51:23.423228    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6" podUID="be45e9ae-2a43-4cd9-bd86-541bc2a97291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 16 12:51:24.422700 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:51:24.422665    2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-9rfz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Sep 16 12:51:24.422879 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:51:24.422720    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6" podUID="be45e9ae-2a43-4cd9-bd86-541bc2a97291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 16 12:51:25.425992 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:51:25.425961    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"
Sep 16 12:51:25.427002 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:51:25.426980    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9rfz6"
Sep 16 12:52:10.842556 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:10.842525    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4slnl"]
Sep 16 12:52:10.846372 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:10.846349    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4slnl"
Sep 16 12:52:10.849141 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:10.849117    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 16 12:52:10.850249 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:10.850223    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 16 12:52:10.850365 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:10.850275    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-q4h2f\""
Sep 16 12:52:10.850365 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:10.850353    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 16 12:52:10.857247 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:10.857226    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4slnl"]
Sep 16 12:52:10.930639 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:10.930606    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80880178-50ec-490f-a609-52efa72dcedc-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4slnl\" (UID: \"80880178-50ec-490f-a609-52efa72dcedc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4slnl"
Sep 16 12:52:10.930639 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:10.930639    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80880178-50ec-490f-a609-52efa72dcedc-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4slnl\" (UID: \"80880178-50ec-490f-a609-52efa72dcedc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4slnl"
Sep 16 12:52:10.930826 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:10.930668    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80880178-50ec-490f-a609-52efa72dcedc-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4slnl\" (UID: \"80880178-50ec-490f-a609-52efa72dcedc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4slnl"
Sep 16 12:52:10.930826 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:10.930766    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80880178-50ec-490f-a609-52efa72dcedc-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4slnl\" (UID: \"80880178-50ec-490f-a609-52efa72dcedc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4slnl"
Sep 16 12:52:10.930826 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:10.930794    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjs6t\" (UniqueName: \"kubernetes.io/projected/80880178-50ec-490f-a609-52efa72dcedc-kube-api-access-hjs6t\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4slnl\" (UID: \"80880178-50ec-490f-a609-52efa72dcedc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4slnl"
Sep 16 12:52:10.930928 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:10.930842    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80880178-50ec-490f-a609-52efa72dcedc-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4slnl\" (UID: \"80880178-50ec-490f-a609-52efa72dcedc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4slnl"
Sep 16 12:52:11.031283 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:11.031252    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80880178-50ec-490f-a609-52efa72dcedc-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4slnl\" (UID: \"80880178-50ec-490f-a609-52efa72dcedc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4slnl"
Sep 16 12:52:11.031283 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:11.031288    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjs6t\" (UniqueName: \"kubernetes.io/projected/80880178-50ec-490f-a609-52efa72dcedc-kube-api-access-hjs6t\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4slnl\" (UID: \"80880178-50ec-490f-a609-52efa72dcedc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4slnl"
Sep 16 12:52:11.031516 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:11.031318    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80880178-50ec-490f-a609-52efa72dcedc-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4slnl\" (UID: \"80880178-50ec-490f-a609-52efa72dcedc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4slnl"
Sep 16 12:52:11.031516 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:11.031367    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80880178-50ec-490f-a609-52efa72dcedc-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4slnl\" (UID: \"80880178-50ec-490f-a609-52efa72dcedc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4slnl"
Sep 16 12:52:11.031516 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:11.031381    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80880178-50ec-490f-a609-52efa72dcedc-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4slnl\" (UID: \"80880178-50ec-490f-a609-52efa72dcedc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4slnl"
Sep 16 12:52:11.031516 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:11.031403    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80880178-50ec-490f-a609-52efa72dcedc-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4slnl\" (UID: \"80880178-50ec-490f-a609-52efa72dcedc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4slnl"
Sep 16 12:52:11.031908 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:11.031883    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80880178-50ec-490f-a609-52efa72dcedc-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4slnl\" (UID: \"80880178-50ec-490f-a609-52efa72dcedc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4slnl"
Sep 16 12:52:11.032114 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:11.031894    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80880178-50ec-490f-a609-52efa72dcedc-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4slnl\" (UID: \"80880178-50ec-490f-a609-52efa72dcedc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4slnl"
Sep 16 12:52:11.032114 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:11.032005    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80880178-50ec-490f-a609-52efa72dcedc-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4slnl\" (UID: \"80880178-50ec-490f-a609-52efa72dcedc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4slnl"
Sep 16 12:52:11.033847 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:11.033822    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80880178-50ec-490f-a609-52efa72dcedc-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4slnl\" (UID: \"80880178-50ec-490f-a609-52efa72dcedc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4slnl"
Sep 16 12:52:11.033972 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:11.033956    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80880178-50ec-490f-a609-52efa72dcedc-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4slnl\" (UID: \"80880178-50ec-490f-a609-52efa72dcedc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4slnl"
Sep 16 12:52:11.039794 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:11.039776    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjs6t\" (UniqueName: \"kubernetes.io/projected/80880178-50ec-490f-a609-52efa72dcedc-kube-api-access-hjs6t\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4slnl\" (UID: \"80880178-50ec-490f-a609-52efa72dcedc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4slnl"
Sep 16 12:52:11.158246 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:11.158154    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4slnl"
Sep 16 12:52:11.436396 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:11.436367    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4slnl"]
Sep 16 12:52:11.438500 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:52:11.438473    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80880178_50ec_490f_a609_52efa72dcedc.slice/crio-2598408d51f7b25dfa02f0131e214d8edb897bd4cfb11af60d2b5b054346263f WatchSource:0}: Error finding container 2598408d51f7b25dfa02f0131e214d8edb897bd4cfb11af60d2b5b054346263f: Status 404 returned error can't find the container with id 2598408d51f7b25dfa02f0131e214d8edb897bd4cfb11af60d2b5b054346263f
Sep 16 12:52:11.440179 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:11.440161    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:52:11.696171 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:11.696089    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4slnl" event={"ID":"80880178-50ec-490f-a609-52efa72dcedc","Type":"ContainerStarted","Data":"2598408d51f7b25dfa02f0131e214d8edb897bd4cfb11af60d2b5b054346263f"}
Sep 16 12:52:12.704340 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:12.704305    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4slnl" event={"ID":"80880178-50ec-490f-a609-52efa72dcedc","Type":"ContainerStarted","Data":"aab0fb5a18d9437d306e93369b5f27c643efab3ee42f1a795bc976fa1816cd4e"}
Sep 16 12:52:12.726485 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:12.726438    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4slnl" podStartSLOduration=1.6809064299999998 podStartE2EDuration="2.726422551s" totalImagesPullingTime="1.045516121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:52:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:52:12.724513484 +0000 UTC m=+839.171746126" watchObservedRunningTime="2026-09-16 12:52:12.726422551 +0000 UTC m=+839.173655192"
Sep 16 12:52:21.159233 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:21.159192    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4slnl"
Sep 16 12:52:21.159660 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:21.159260    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4slnl"
Sep 16 12:52:21.172073 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:21.172049    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4slnl"
Sep 16 12:52:21.469660 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:21.467207    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf"]
Sep 16 12:52:21.471435 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:21.471411    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf"
Sep 16 12:52:21.474687 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:21.474670    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 16 12:52:21.481702 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:21.481681    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf"]
Sep 16 12:52:21.530567 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:21.530537    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e54d5f1e-0503-4b28-88f6-53cd5ac6062e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf\" (UID: \"e54d5f1e-0503-4b28-88f6-53cd5ac6062e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf"
Sep 16 12:52:21.530754 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:21.530575    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e54d5f1e-0503-4b28-88f6-53cd5ac6062e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf\" (UID: \"e54d5f1e-0503-4b28-88f6-53cd5ac6062e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf"
Sep 16 12:52:21.530754 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:21.530596    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e54d5f1e-0503-4b28-88f6-53cd5ac6062e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf\" (UID: \"e54d5f1e-0503-4b28-88f6-53cd5ac6062e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf"
Sep 16 12:52:21.530754 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:21.530626    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx6z4\" (UniqueName: \"kubernetes.io/projected/e54d5f1e-0503-4b28-88f6-53cd5ac6062e-kube-api-access-sx6z4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf\" (UID: \"e54d5f1e-0503-4b28-88f6-53cd5ac6062e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf"
Sep 16 12:52:21.530754 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:21.530650    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e54d5f1e-0503-4b28-88f6-53cd5ac6062e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf\" (UID: \"e54d5f1e-0503-4b28-88f6-53cd5ac6062e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf"
Sep 16 12:52:21.530754 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:21.530672    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e54d5f1e-0503-4b28-88f6-53cd5ac6062e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf\" (UID: \"e54d5f1e-0503-4b28-88f6-53cd5ac6062e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf"
Sep 16 12:52:21.631244 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:21.631207    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e54d5f1e-0503-4b28-88f6-53cd5ac6062e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf\" (UID: \"e54d5f1e-0503-4b28-88f6-53cd5ac6062e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf"
Sep 16 12:52:21.631486 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:21.631258    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e54d5f1e-0503-4b28-88f6-53cd5ac6062e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf\" (UID: \"e54d5f1e-0503-4b28-88f6-53cd5ac6062e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf"
Sep 16 12:52:21.631486 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:21.631354    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e54d5f1e-0503-4b28-88f6-53cd5ac6062e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf\" (UID: \"e54d5f1e-0503-4b28-88f6-53cd5ac6062e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf"
Sep 16 12:52:21.631486 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:21.631375    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e54d5f1e-0503-4b28-88f6-53cd5ac6062e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf\" (UID: \"e54d5f1e-0503-4b28-88f6-53cd5ac6062e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf"
Sep 16 12:52:21.631665 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:21.631401    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e54d5f1e-0503-4b28-88f6-53cd5ac6062e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf\" (UID: \"e54d5f1e-0503-4b28-88f6-53cd5ac6062e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf"
Sep 16 12:52:21.631665 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:21.631544    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx6z4\" (UniqueName: \"kubernetes.io/projected/e54d5f1e-0503-4b28-88f6-53cd5ac6062e-kube-api-access-sx6z4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf\" (UID: \"e54d5f1e-0503-4b28-88f6-53cd5ac6062e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf"
Sep 16 12:52:21.631775 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:21.631701    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e54d5f1e-0503-4b28-88f6-53cd5ac6062e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf\" (UID: \"e54d5f1e-0503-4b28-88f6-53cd5ac6062e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf"
Sep 16 12:52:21.632168 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:21.632105    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e54d5f1e-0503-4b28-88f6-53cd5ac6062e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf\" (UID: \"e54d5f1e-0503-4b28-88f6-53cd5ac6062e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf"
Sep 16 12:52:21.632168 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:21.632115    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e54d5f1e-0503-4b28-88f6-53cd5ac6062e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf\" (UID: \"e54d5f1e-0503-4b28-88f6-53cd5ac6062e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf"
Sep 16 12:52:21.634590 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:21.634566    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e54d5f1e-0503-4b28-88f6-53cd5ac6062e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf\" (UID: \"e54d5f1e-0503-4b28-88f6-53cd5ac6062e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf"
Sep 16 12:52:21.634915 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:21.634892    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e54d5f1e-0503-4b28-88f6-53cd5ac6062e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf\" (UID: \"e54d5f1e-0503-4b28-88f6-53cd5ac6062e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf"
Sep 16 12:52:21.640709 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:21.640682    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx6z4\" (UniqueName: \"kubernetes.io/projected/e54d5f1e-0503-4b28-88f6-53cd5ac6062e-kube-api-access-sx6z4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf\" (UID: \"e54d5f1e-0503-4b28-88f6-53cd5ac6062e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf"
Sep 16 12:52:21.760404 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:21.760375    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4slnl"
Sep 16 12:52:21.781539 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:21.781514    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf"
Sep 16 12:52:22.061506 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:22.061482    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf"]
Sep 16 12:52:22.063525 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:52:22.063486    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode54d5f1e_0503_4b28_88f6_53cd5ac6062e.slice/crio-8c49b2c996a4051a260a2d01f9156d2ab893421d60842e17fec5a81cf2a4b58a WatchSource:0}: Error finding container 8c49b2c996a4051a260a2d01f9156d2ab893421d60842e17fec5a81cf2a4b58a: Status 404 returned error can't find the container with id 8c49b2c996a4051a260a2d01f9156d2ab893421d60842e17fec5a81cf2a4b58a
Sep 16 12:52:22.754972 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:22.754931    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf" event={"ID":"e54d5f1e-0503-4b28-88f6-53cd5ac6062e","Type":"ContainerStarted","Data":"7717b9d6e1988aa3f43fcd857e4f760eef6c426ff006010c7746cc675afd9419"}
Sep 16 12:52:22.755462 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:22.755052    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf" event={"ID":"e54d5f1e-0503-4b28-88f6-53cd5ac6062e","Type":"ContainerStarted","Data":"8c49b2c996a4051a260a2d01f9156d2ab893421d60842e17fec5a81cf2a4b58a"}
Sep 16 12:52:22.779727 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:22.779677    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf" podStartSLOduration=1.573321418 podStartE2EDuration="1.779662669s" totalImagesPullingTime="206.341251ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:52:22.778542261 +0000 UTC m=+849.225774904" watchObservedRunningTime="2026-09-16 12:52:22.779662669 +0000 UTC m=+849.226895310"
Sep 16 12:52:31.782271 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:31.782236    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf"
Sep 16 12:52:31.782655 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:31.782291    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf"
Sep 16 12:52:31.798509 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:31.798483    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf"
Sep 16 12:52:31.811262 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:31.811240    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf"
Sep 16 12:52:38.944501 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:38.944470    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-bvp5c"]
Sep 16 12:52:38.948792 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:38.948770    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bvp5c"
Sep 16 12:52:38.951483 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:38.951458    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 16 12:52:38.958418 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:38.958394    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-bvp5c"]
Sep 16 12:52:39.097941 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:39.097908    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8010ee2c-f62f-4b5a-a729-e26019b3364d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bvp5c\" (UID: \"8010ee2c-f62f-4b5a-a729-e26019b3364d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bvp5c"
Sep 16 12:52:39.098114 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:39.097975    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8010ee2c-f62f-4b5a-a729-e26019b3364d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bvp5c\" (UID: \"8010ee2c-f62f-4b5a-a729-e26019b3364d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bvp5c"
Sep 16 12:52:39.098114 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:39.098012    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8010ee2c-f62f-4b5a-a729-e26019b3364d-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bvp5c\" (UID: \"8010ee2c-f62f-4b5a-a729-e26019b3364d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bvp5c"
Sep 16 12:52:39.098114 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:39.098054    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8010ee2c-f62f-4b5a-a729-e26019b3364d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bvp5c\" (UID: \"8010ee2c-f62f-4b5a-a729-e26019b3364d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bvp5c"
Sep 16 12:52:39.098223 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:39.098132    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghtrn\" (UniqueName: \"kubernetes.io/projected/8010ee2c-f62f-4b5a-a729-e26019b3364d-kube-api-access-ghtrn\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bvp5c\" (UID: \"8010ee2c-f62f-4b5a-a729-e26019b3364d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bvp5c"
Sep 16 12:52:39.098223 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:39.098178    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8010ee2c-f62f-4b5a-a729-e26019b3364d-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bvp5c\" (UID: \"8010ee2c-f62f-4b5a-a729-e26019b3364d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bvp5c"
Sep 16 12:52:39.198802 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:39.198712    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8010ee2c-f62f-4b5a-a729-e26019b3364d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bvp5c\" (UID: \"8010ee2c-f62f-4b5a-a729-e26019b3364d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bvp5c"
Sep 16 12:52:39.198802 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:39.198776    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8010ee2c-f62f-4b5a-a729-e26019b3364d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bvp5c\" (UID: \"8010ee2c-f62f-4b5a-a729-e26019b3364d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bvp5c"
Sep 16 12:52:39.199014 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:39.198807    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8010ee2c-f62f-4b5a-a729-e26019b3364d-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bvp5c\" (UID: \"8010ee2c-f62f-4b5a-a729-e26019b3364d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bvp5c"
Sep 16 12:52:39.199014 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:39.198830    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8010ee2c-f62f-4b5a-a729-e26019b3364d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bvp5c\" (UID: \"8010ee2c-f62f-4b5a-a729-e26019b3364d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bvp5c"
Sep 16 12:52:39.199014 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:39.198889    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghtrn\" (UniqueName: \"kubernetes.io/projected/8010ee2c-f62f-4b5a-a729-e26019b3364d-kube-api-access-ghtrn\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bvp5c\" (UID: \"8010ee2c-f62f-4b5a-a729-e26019b3364d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bvp5c"
Sep 16 12:52:39.199014 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:39.198946    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8010ee2c-f62f-4b5a-a729-e26019b3364d-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bvp5c\" (UID: \"8010ee2c-f62f-4b5a-a729-e26019b3364d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bvp5c"
Sep 16 12:52:39.199359 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:39.199330    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8010ee2c-f62f-4b5a-a729-e26019b3364d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bvp5c\" (UID: \"8010ee2c-f62f-4b5a-a729-e26019b3364d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bvp5c"
Sep 16 12:52:39.199640 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:39.199382    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8010ee2c-f62f-4b5a-a729-e26019b3364d-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bvp5c\" (UID: \"8010ee2c-f62f-4b5a-a729-e26019b3364d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bvp5c"
Sep 16 12:52:39.199640 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:39.199448    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8010ee2c-f62f-4b5a-a729-e26019b3364d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bvp5c\" (UID: \"8010ee2c-f62f-4b5a-a729-e26019b3364d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bvp5c"
Sep 16 12:52:39.201312 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:39.201278    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8010ee2c-f62f-4b5a-a729-e26019b3364d-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bvp5c\" (UID: \"8010ee2c-f62f-4b5a-a729-e26019b3364d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bvp5c"
Sep 16 12:52:39.201435 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:39.201421    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8010ee2c-f62f-4b5a-a729-e26019b3364d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bvp5c\" (UID: \"8010ee2c-f62f-4b5a-a729-e26019b3364d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bvp5c"
Sep 16 12:52:39.210288 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:39.210270    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghtrn\" (UniqueName: \"kubernetes.io/projected/8010ee2c-f62f-4b5a-a729-e26019b3364d-kube-api-access-ghtrn\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bvp5c\" (UID: \"8010ee2c-f62f-4b5a-a729-e26019b3364d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bvp5c"
Sep 16 12:52:39.261874 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:39.261840    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bvp5c"
Sep 16 12:52:39.540573 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:39.540540    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-bvp5c"]
Sep 16 12:52:39.543176 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:52:39.543151    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8010ee2c_f62f_4b5a_a729_e26019b3364d.slice/crio-138706b75ef7dcc17f7815f4f5f8844d6abc875f7a44d6616cdd474ead0a8632 WatchSource:0}: Error finding container 138706b75ef7dcc17f7815f4f5f8844d6abc875f7a44d6616cdd474ead0a8632: Status 404 returned error can't find the container with id 138706b75ef7dcc17f7815f4f5f8844d6abc875f7a44d6616cdd474ead0a8632
Sep 16 12:52:39.841559 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:39.841515    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bvp5c" event={"ID":"8010ee2c-f62f-4b5a-a729-e26019b3364d","Type":"ContainerStarted","Data":"898ff02d91f74aa016c76c86185639877cfb7bed9bd80f92f7a5b947187a40cb"}
Sep 16 12:52:39.841733 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:39.841579    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bvp5c" event={"ID":"8010ee2c-f62f-4b5a-a729-e26019b3364d","Type":"ContainerStarted","Data":"138706b75ef7dcc17f7815f4f5f8844d6abc875f7a44d6616cdd474ead0a8632"}
Sep 16 12:52:39.862144 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:39.862094    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bvp5c" podStartSLOduration=1.66958732 podStartE2EDuration="1.862075598s" totalImagesPullingTime="192.488278ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:52:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:52:39.860676545 +0000 UTC m=+866.307909201" watchObservedRunningTime="2026-09-16 12:52:39.862075598 +0000 UTC m=+866.309308243"
Sep 16 12:52:49.262964 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:49.262928    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bvp5c"
Sep 16 12:52:49.263433 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:49.262994    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bvp5c"
Sep 16 12:52:49.276010 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:49.275986    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bvp5c"
Sep 16 12:52:49.901955 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:52:49.901924    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bvp5c"
Sep 16 12:53:14.064738 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:53:14.064656    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vbt8d_0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174/console-operator/1.log"
Sep 16 12:53:14.066847 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:53:14.066825    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vbt8d_0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174/console-operator/1.log"
Sep 16 12:54:14.611316 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:14.611280    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5fcdd846cb-f5q2s"]
Sep 16 12:54:14.612010 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:14.611510    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5fcdd846cb-f5q2s" podUID="299d7f29-6a45-479a-a225-04de47548ab1" containerName="authorino" containerID="cri-o://1a84b8c29ccf7b1f239286bf9eec1685060fc7098cda08db7b4f48d2f60e38e9" gracePeriod=30
Sep 16 12:54:14.861381 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:14.861324    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fcdd846cb-f5q2s"
Sep 16 12:54:14.944503 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:14.944470    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/299d7f29-6a45-479a-a225-04de47548ab1-tls-cert\") pod \"299d7f29-6a45-479a-a225-04de47548ab1\" (UID: \"299d7f29-6a45-479a-a225-04de47548ab1\") "
Sep 16 12:54:14.944681 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:14.944525    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-294mt\" (UniqueName: \"kubernetes.io/projected/299d7f29-6a45-479a-a225-04de47548ab1-kube-api-access-294mt\") pod \"299d7f29-6a45-479a-a225-04de47548ab1\" (UID: \"299d7f29-6a45-479a-a225-04de47548ab1\") "
Sep 16 12:54:14.946745 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:14.946712    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/299d7f29-6a45-479a-a225-04de47548ab1-kube-api-access-294mt" (OuterVolumeSpecName: "kube-api-access-294mt") pod "299d7f29-6a45-479a-a225-04de47548ab1" (UID: "299d7f29-6a45-479a-a225-04de47548ab1"). InnerVolumeSpecName "kube-api-access-294mt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:54:14.955347 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:14.955323    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/299d7f29-6a45-479a-a225-04de47548ab1-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "299d7f29-6a45-479a-a225-04de47548ab1" (UID: "299d7f29-6a45-479a-a225-04de47548ab1"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:54:15.045175 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:15.045147    2577 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/299d7f29-6a45-479a-a225-04de47548ab1-tls-cert\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:54:15.045175 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:15.045170    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-294mt\" (UniqueName: \"kubernetes.io/projected/299d7f29-6a45-479a-a225-04de47548ab1-kube-api-access-294mt\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:54:15.319462 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:15.319422    2577 generic.go:356] "Generic (PLEG): container finished" podID="299d7f29-6a45-479a-a225-04de47548ab1" containerID="1a84b8c29ccf7b1f239286bf9eec1685060fc7098cda08db7b4f48d2f60e38e9" exitCode=0
Sep 16 12:54:15.319642 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:15.319474    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fcdd846cb-f5q2s"
Sep 16 12:54:15.319642 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:15.319510    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fcdd846cb-f5q2s" event={"ID":"299d7f29-6a45-479a-a225-04de47548ab1","Type":"ContainerDied","Data":"1a84b8c29ccf7b1f239286bf9eec1685060fc7098cda08db7b4f48d2f60e38e9"}
Sep 16 12:54:15.319642 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:15.319559    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fcdd846cb-f5q2s" event={"ID":"299d7f29-6a45-479a-a225-04de47548ab1","Type":"ContainerDied","Data":"371e61c33e78e72b9a0dc1cdae373422ca31564aa73b9ae9688f2d3eae7f0a08"}
Sep 16 12:54:15.319642 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:15.319582    2577 scope.go:117] "RemoveContainer" containerID="1a84b8c29ccf7b1f239286bf9eec1685060fc7098cda08db7b4f48d2f60e38e9"
Sep 16 12:54:15.328912 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:15.328895    2577 scope.go:117] "RemoveContainer" containerID="1a84b8c29ccf7b1f239286bf9eec1685060fc7098cda08db7b4f48d2f60e38e9"
Sep 16 12:54:15.329201 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:54:15.329182    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a84b8c29ccf7b1f239286bf9eec1685060fc7098cda08db7b4f48d2f60e38e9\": container with ID starting with 1a84b8c29ccf7b1f239286bf9eec1685060fc7098cda08db7b4f48d2f60e38e9 not found: ID does not exist" containerID="1a84b8c29ccf7b1f239286bf9eec1685060fc7098cda08db7b4f48d2f60e38e9"
Sep 16 12:54:15.329262 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:15.329211    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a84b8c29ccf7b1f239286bf9eec1685060fc7098cda08db7b4f48d2f60e38e9"} err="failed to get container status \"1a84b8c29ccf7b1f239286bf9eec1685060fc7098cda08db7b4f48d2f60e38e9\": rpc error: code = NotFound desc = could not find container \"1a84b8c29ccf7b1f239286bf9eec1685060fc7098cda08db7b4f48d2f60e38e9\": container with ID starting with 1a84b8c29ccf7b1f239286bf9eec1685060fc7098cda08db7b4f48d2f60e38e9 not found: ID does not exist"
Sep 16 12:54:15.346154 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:15.346126    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5fcdd846cb-f5q2s"]
Sep 16 12:54:15.349598 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:15.349573    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5fcdd846cb-f5q2s"]
Sep 16 12:54:16.055383 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:16.055347    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="299d7f29-6a45-479a-a225-04de47548ab1" path="/var/lib/kubelet/pods/299d7f29-6a45-479a-a225-04de47548ab1/volumes"
Sep 16 12:54:38.634974 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.634899    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"]
Sep 16 12:54:38.635428 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.635302    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="299d7f29-6a45-479a-a225-04de47548ab1" containerName="authorino"
Sep 16 12:54:38.635428 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.635314    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="299d7f29-6a45-479a-a225-04de47548ab1" containerName="authorino"
Sep 16 12:54:38.635428 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.635398    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="299d7f29-6a45-479a-a225-04de47548ab1" containerName="authorino"
Sep 16 12:54:38.640128 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.640105    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"
Sep 16 12:54:38.643001 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.642981    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-7d0256-openshift-default-dockercfg-zphj6\""
Sep 16 12:54:38.652441 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.652421    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"]
Sep 16 12:54:38.659763 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.659738    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99b70637-81ce-4503-bc3d-de5dc69e7b97-istio-podinfo\") pod \"e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz\" (UID: \"99b70637-81ce-4503-bc3d-de5dc69e7b97\") " pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"
Sep 16 12:54:38.659870 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.659796    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99b70637-81ce-4503-bc3d-de5dc69e7b97-istio-token\") pod \"e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz\" (UID: \"99b70637-81ce-4503-bc3d-de5dc69e7b97\") " pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"
Sep 16 12:54:38.659870 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.659825    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99b70637-81ce-4503-bc3d-de5dc69e7b97-istiod-ca-cert\") pod \"e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz\" (UID: \"99b70637-81ce-4503-bc3d-de5dc69e7b97\") " pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"
Sep 16 12:54:38.659870 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.659859    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbqpp\" (UniqueName: \"kubernetes.io/projected/99b70637-81ce-4503-bc3d-de5dc69e7b97-kube-api-access-qbqpp\") pod \"e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz\" (UID: \"99b70637-81ce-4503-bc3d-de5dc69e7b97\") " pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"
Sep 16 12:54:38.660050 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.659886    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99b70637-81ce-4503-bc3d-de5dc69e7b97-istio-data\") pod \"e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz\" (UID: \"99b70637-81ce-4503-bc3d-de5dc69e7b97\") " pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"
Sep 16 12:54:38.660050 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.659926    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99b70637-81ce-4503-bc3d-de5dc69e7b97-workload-certs\") pod \"e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz\" (UID: \"99b70637-81ce-4503-bc3d-de5dc69e7b97\") " pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"
Sep 16 12:54:38.660050 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.659949    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99b70637-81ce-4503-bc3d-de5dc69e7b97-istio-envoy\") pod \"e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz\" (UID: \"99b70637-81ce-4503-bc3d-de5dc69e7b97\") " pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"
Sep 16 12:54:38.660200 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.660074    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b70637-81ce-4503-bc3d-de5dc69e7b97-workload-socket\") pod \"e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz\" (UID: \"99b70637-81ce-4503-bc3d-de5dc69e7b97\") " pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"
Sep 16 12:54:38.660200 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.660104    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b70637-81ce-4503-bc3d-de5dc69e7b97-credential-socket\") pod \"e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz\" (UID: \"99b70637-81ce-4503-bc3d-de5dc69e7b97\") " pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"
Sep 16 12:54:38.710113 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.710077    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"]
Sep 16 12:54:38.713607 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.713583    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"
Sep 16 12:54:38.716356 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.716333    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-490797-openshift-default-dockercfg-n72qf\""
Sep 16 12:54:38.725335 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.725315    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"]
Sep 16 12:54:38.760809 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.760769    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/462e0727-223b-4217-af33-aae07198ca07-istiod-ca-cert\") pod \"e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs\" (UID: \"462e0727-223b-4217-af33-aae07198ca07\") " pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"
Sep 16 12:54:38.760809 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.760819    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99b70637-81ce-4503-bc3d-de5dc69e7b97-istio-token\") pod \"e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz\" (UID: \"99b70637-81ce-4503-bc3d-de5dc69e7b97\") " pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"
Sep 16 12:54:38.761009 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.760879    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99b70637-81ce-4503-bc3d-de5dc69e7b97-istio-podinfo\") pod \"e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz\" (UID: \"99b70637-81ce-4503-bc3d-de5dc69e7b97\") " pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"
Sep 16 12:54:38.761009 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.760922    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/462e0727-223b-4217-af33-aae07198ca07-workload-socket\") pod \"e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs\" (UID: \"462e0727-223b-4217-af33-aae07198ca07\") " pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"
Sep 16 12:54:38.761009 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.760946    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99b70637-81ce-4503-bc3d-de5dc69e7b97-istio-data\") pod \"e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz\" (UID: \"99b70637-81ce-4503-bc3d-de5dc69e7b97\") " pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"
Sep 16 12:54:38.761009 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.760969    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfwr7\" (UniqueName: \"kubernetes.io/projected/462e0727-223b-4217-af33-aae07198ca07-kube-api-access-zfwr7\") pod \"e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs\" (UID: \"462e0727-223b-4217-af33-aae07198ca07\") " pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"
Sep 16 12:54:38.761009 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.760995    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99b70637-81ce-4503-bc3d-de5dc69e7b97-istiod-ca-cert\") pod \"e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz\" (UID: \"99b70637-81ce-4503-bc3d-de5dc69e7b97\") " pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"
Sep 16 12:54:38.761268 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.761121    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b70637-81ce-4503-bc3d-de5dc69e7b97-credential-socket\") pod \"e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz\" (UID: \"99b70637-81ce-4503-bc3d-de5dc69e7b97\") " pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"
Sep 16 12:54:38.761268 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.761202    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/462e0727-223b-4217-af33-aae07198ca07-workload-certs\") pod \"e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs\" (UID: \"462e0727-223b-4217-af33-aae07198ca07\") " pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"
Sep 16 12:54:38.761268 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.761239    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/462e0727-223b-4217-af33-aae07198ca07-istio-token\") pod \"e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs\" (UID: \"462e0727-223b-4217-af33-aae07198ca07\") " pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"
Sep 16 12:54:38.761425 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.761280    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/462e0727-223b-4217-af33-aae07198ca07-istio-envoy\") pod \"e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs\" (UID: \"462e0727-223b-4217-af33-aae07198ca07\") " pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"
Sep 16 12:54:38.761425 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.761371    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/462e0727-223b-4217-af33-aae07198ca07-istio-podinfo\") pod \"e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs\" (UID: \"462e0727-223b-4217-af33-aae07198ca07\") " pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"
Sep 16 12:54:38.761525 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.761442    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbqpp\" (UniqueName: \"kubernetes.io/projected/99b70637-81ce-4503-bc3d-de5dc69e7b97-kube-api-access-qbqpp\") pod \"e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz\" (UID: \"99b70637-81ce-4503-bc3d-de5dc69e7b97\") " pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"
Sep 16 12:54:38.761525 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.761474    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b70637-81ce-4503-bc3d-de5dc69e7b97-credential-socket\") pod \"e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz\" (UID: \"99b70637-81ce-4503-bc3d-de5dc69e7b97\") " pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"
Sep 16 12:54:38.761633 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.761534    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99b70637-81ce-4503-bc3d-de5dc69e7b97-workload-certs\") pod \"e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz\" (UID: \"99b70637-81ce-4503-bc3d-de5dc69e7b97\") " pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"
Sep 16 12:54:38.761633 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.761544    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99b70637-81ce-4503-bc3d-de5dc69e7b97-istio-data\") pod \"e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz\" (UID: \"99b70637-81ce-4503-bc3d-de5dc69e7b97\") " pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"
Sep 16 12:54:38.761633 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.761562    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99b70637-81ce-4503-bc3d-de5dc69e7b97-istio-envoy\") pod \"e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz\" (UID: \"99b70637-81ce-4503-bc3d-de5dc69e7b97\") " pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"
Sep 16 12:54:38.761633 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.761595    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/462e0727-223b-4217-af33-aae07198ca07-istio-data\") pod \"e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs\" (UID: \"462e0727-223b-4217-af33-aae07198ca07\") " pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"
Sep 16 12:54:38.761820 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.761630    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/462e0727-223b-4217-af33-aae07198ca07-credential-socket\") pod \"e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs\" (UID: \"462e0727-223b-4217-af33-aae07198ca07\") " pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"
Sep 16 12:54:38.761820 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.761662    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b70637-81ce-4503-bc3d-de5dc69e7b97-workload-socket\") pod \"e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz\" (UID: \"99b70637-81ce-4503-bc3d-de5dc69e7b97\") " pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"
Sep 16 12:54:38.761820 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.761774    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99b70637-81ce-4503-bc3d-de5dc69e7b97-istiod-ca-cert\") pod \"e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz\" (UID: \"99b70637-81ce-4503-bc3d-de5dc69e7b97\") " pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"
Sep 16 12:54:38.761820 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.761803    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99b70637-81ce-4503-bc3d-de5dc69e7b97-workload-certs\") pod \"e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz\" (UID: \"99b70637-81ce-4503-bc3d-de5dc69e7b97\") " pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"
Sep 16 12:54:38.762202 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.762085    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b70637-81ce-4503-bc3d-de5dc69e7b97-workload-socket\") pod \"e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz\" (UID: \"99b70637-81ce-4503-bc3d-de5dc69e7b97\") " pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"
Sep 16 12:54:38.763820 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.763779    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99b70637-81ce-4503-bc3d-de5dc69e7b97-istio-podinfo\") pod \"e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz\" (UID: \"99b70637-81ce-4503-bc3d-de5dc69e7b97\") " pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"
Sep 16 12:54:38.763994 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.763967    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99b70637-81ce-4503-bc3d-de5dc69e7b97-istio-envoy\") pod \"e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz\" (UID: \"99b70637-81ce-4503-bc3d-de5dc69e7b97\") " pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"
Sep 16 12:54:38.773739 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.773713    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbqpp\" (UniqueName: \"kubernetes.io/projected/99b70637-81ce-4503-bc3d-de5dc69e7b97-kube-api-access-qbqpp\") pod \"e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz\" (UID: \"99b70637-81ce-4503-bc3d-de5dc69e7b97\") " pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"
Sep 16 12:54:38.774173 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.774141    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99b70637-81ce-4503-bc3d-de5dc69e7b97-istio-token\") pod \"e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz\" (UID: \"99b70637-81ce-4503-bc3d-de5dc69e7b97\") " pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"
Sep 16 12:54:38.862806 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.862771    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/462e0727-223b-4217-af33-aae07198ca07-istio-podinfo\") pod \"e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs\" (UID: \"462e0727-223b-4217-af33-aae07198ca07\") " pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"
Sep 16 12:54:38.862991 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.862822    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/462e0727-223b-4217-af33-aae07198ca07-istio-data\") pod \"e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs\" (UID: \"462e0727-223b-4217-af33-aae07198ca07\") " pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"
Sep 16 12:54:38.862991 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.862848    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/462e0727-223b-4217-af33-aae07198ca07-credential-socket\") pod \"e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs\" (UID: \"462e0727-223b-4217-af33-aae07198ca07\") " pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"
Sep 16 12:54:38.862991 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.862886    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/462e0727-223b-4217-af33-aae07198ca07-istiod-ca-cert\") pod \"e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs\" (UID: \"462e0727-223b-4217-af33-aae07198ca07\") " pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"
Sep 16 12:54:38.862991 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.862941    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/462e0727-223b-4217-af33-aae07198ca07-workload-socket\") pod \"e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs\" (UID: \"462e0727-223b-4217-af33-aae07198ca07\") " pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"
Sep 16 12:54:38.862991 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.862975    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfwr7\" (UniqueName: \"kubernetes.io/projected/462e0727-223b-4217-af33-aae07198ca07-kube-api-access-zfwr7\") pod \"e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs\" (UID: \"462e0727-223b-4217-af33-aae07198ca07\") " pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"
Sep 16 12:54:38.863315 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.863053    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/462e0727-223b-4217-af33-aae07198ca07-workload-certs\") pod \"e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs\" (UID: \"462e0727-223b-4217-af33-aae07198ca07\") " pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"
Sep 16 12:54:38.863315 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.863086    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/462e0727-223b-4217-af33-aae07198ca07-istio-token\") pod \"e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs\" (UID: \"462e0727-223b-4217-af33-aae07198ca07\") " pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"
Sep 16 12:54:38.863315 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.863120    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/462e0727-223b-4217-af33-aae07198ca07-istio-envoy\") pod \"e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs\" (UID: \"462e0727-223b-4217-af33-aae07198ca07\") " pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"
Sep 16 12:54:38.863477 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.863334    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/462e0727-223b-4217-af33-aae07198ca07-credential-socket\") pod \"e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs\" (UID: \"462e0727-223b-4217-af33-aae07198ca07\") " pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"
Sep 16 12:54:38.863477 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.863426    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/462e0727-223b-4217-af33-aae07198ca07-workload-socket\") pod \"e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs\" (UID: \"462e0727-223b-4217-af33-aae07198ca07\") " pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"
Sep 16 12:54:38.863477 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.863465    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/462e0727-223b-4217-af33-aae07198ca07-workload-certs\") pod \"e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs\" (UID: \"462e0727-223b-4217-af33-aae07198ca07\") " pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"
Sep 16 12:54:38.863715 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.863692    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/462e0727-223b-4217-af33-aae07198ca07-istiod-ca-cert\") pod \"e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs\" (UID: \"462e0727-223b-4217-af33-aae07198ca07\") " pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"
Sep 16 12:54:38.863824 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.863804    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/462e0727-223b-4217-af33-aae07198ca07-istio-data\") pod \"e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs\" (UID: \"462e0727-223b-4217-af33-aae07198ca07\") " pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"
Sep 16 12:54:38.865473 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.865448    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/462e0727-223b-4217-af33-aae07198ca07-istio-envoy\") pod \"e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs\" (UID: \"462e0727-223b-4217-af33-aae07198ca07\") " pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"
Sep 16 12:54:38.865564 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.865500    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/462e0727-223b-4217-af33-aae07198ca07-istio-podinfo\") pod \"e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs\" (UID: \"462e0727-223b-4217-af33-aae07198ca07\") " pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"
Sep 16 12:54:38.871718 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.871690    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/462e0727-223b-4217-af33-aae07198ca07-istio-token\") pod \"e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs\" (UID: \"462e0727-223b-4217-af33-aae07198ca07\") " pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"
Sep 16 12:54:38.871809 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.871748    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfwr7\" (UniqueName: \"kubernetes.io/projected/462e0727-223b-4217-af33-aae07198ca07-kube-api-access-zfwr7\") pod \"e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs\" (UID: \"462e0727-223b-4217-af33-aae07198ca07\") " pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"
Sep 16 12:54:38.951624 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:38.951538    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"
Sep 16 12:54:39.025654 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:39.025625    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"
Sep 16 12:54:39.233400 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:39.233379    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"]
Sep 16 12:54:39.235741 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:54:39.235717    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99b70637_81ce_4503_bc3d_de5dc69e7b97.slice/crio-2ac85d1a96678df1933082b4c4941afccd308594341802c5a8944bf5442e62e5 WatchSource:0}: Error finding container 2ac85d1a96678df1933082b4c4941afccd308594341802c5a8944bf5442e62e5: Status 404 returned error can't find the container with id 2ac85d1a96678df1933082b4c4941afccd308594341802c5a8944bf5442e62e5
Sep 16 12:54:39.237798 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:39.237768    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:54:39.237875 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:39.237829    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:54:39.237875 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:39.237859    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:54:39.309973 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:39.309947    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"]
Sep 16 12:54:39.310731 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:54:39.310700    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod462e0727_223b_4217_af33_aae07198ca07.slice/crio-1c95c22c973d679355f93354ba45c15164ad7a07856b96e424e1e9065a4e2395 WatchSource:0}: Error finding container 1c95c22c973d679355f93354ba45c15164ad7a07856b96e424e1e9065a4e2395: Status 404 returned error can't find the container with id 1c95c22c973d679355f93354ba45c15164ad7a07856b96e424e1e9065a4e2395
Sep 16 12:54:39.312918 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:39.312893    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:54:39.312981 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:39.312950    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:54:39.313052 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:39.312979    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:54:39.452943 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:39.452897    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz" event={"ID":"99b70637-81ce-4503-bc3d-de5dc69e7b97","Type":"ContainerStarted","Data":"472837c031edae40d3d5c05ad30b7ed63e00da96d92d905048f17581c59547d8"}
Sep 16 12:54:39.453166 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:39.452975    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz" event={"ID":"99b70637-81ce-4503-bc3d-de5dc69e7b97","Type":"ContainerStarted","Data":"2ac85d1a96678df1933082b4c4941afccd308594341802c5a8944bf5442e62e5"}
Sep 16 12:54:39.454787 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:39.454756    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs" event={"ID":"462e0727-223b-4217-af33-aae07198ca07","Type":"ContainerStarted","Data":"c8f872f61ddab2e36fd291728ca8cf8f7faa954725d521c6dc6c328eb7775045"}
Sep 16 12:54:39.454925 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:39.454805    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs" event={"ID":"462e0727-223b-4217-af33-aae07198ca07","Type":"ContainerStarted","Data":"1c95c22c973d679355f93354ba45c15164ad7a07856b96e424e1e9065a4e2395"}
Sep 16 12:54:39.477610 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:39.477547    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz" podStartSLOduration=1.47752589 podStartE2EDuration="1.47752589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:54:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:54:39.475162792 +0000 UTC m=+985.922395447" watchObservedRunningTime="2026-09-16 12:54:39.47752589 +0000 UTC m=+985.924758532"
Sep 16 12:54:39.505745 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:39.505693    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs" podStartSLOduration=1.50567708 podStartE2EDuration="1.50567708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:54:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:54:39.503764624 +0000 UTC m=+985.950997268" watchObservedRunningTime="2026-09-16 12:54:39.50567708 +0000 UTC m=+985.952909797"
Sep 16 12:54:39.952019 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:39.951931    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"
Sep 16 12:54:40.026456 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:40.026426    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"
Sep 16 12:54:40.031605 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:40.031577    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"
Sep 16 12:54:40.292126 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:40.292090    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7553-c9c26a"]
Sep 16 12:54:40.295546 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:40.295524    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7553-c9c26a"
Sep 16 12:54:40.298133 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:40.298106    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-z9lzx\""
Sep 16 12:54:40.300314 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:40.300288    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7553-c9c26a"]
Sep 16 12:54:40.377713 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:40.377679    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfzwn\" (UniqueName: \"kubernetes.io/projected/697fa6b7-876f-4dbd-a03c-2913e6804adc-kube-api-access-cfzwn\") pod \"test-curl-7553-c9c26a\" (UID: \"697fa6b7-876f-4dbd-a03c-2913e6804adc\") " pod="opendatahub/test-curl-7553-c9c26a"
Sep 16 12:54:40.459863 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:40.459827    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"
Sep 16 12:54:40.460636 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:40.460617    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"
Sep 16 12:54:40.478644 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:40.478610    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfzwn\" (UniqueName: \"kubernetes.io/projected/697fa6b7-876f-4dbd-a03c-2913e6804adc-kube-api-access-cfzwn\") pod \"test-curl-7553-c9c26a\" (UID: \"697fa6b7-876f-4dbd-a03c-2913e6804adc\") " pod="opendatahub/test-curl-7553-c9c26a"
Sep 16 12:54:40.487928 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:40.487314    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfzwn\" (UniqueName: \"kubernetes.io/projected/697fa6b7-876f-4dbd-a03c-2913e6804adc-kube-api-access-cfzwn\") pod \"test-curl-7553-c9c26a\" (UID: \"697fa6b7-876f-4dbd-a03c-2913e6804adc\") " pod="opendatahub/test-curl-7553-c9c26a"
Sep 16 12:54:40.607486 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:40.607407    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7553-c9c26a"
Sep 16 12:54:40.883827 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:40.883799    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7553-c9c26a"]
Sep 16 12:54:40.885772 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:54:40.885745    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod697fa6b7_876f_4dbd_a03c_2913e6804adc.slice/crio-44443e303df2fcfdf73eec50a3ea2b8e0f28ad89b507af2c0ce9f541126560a8 WatchSource:0}: Error finding container 44443e303df2fcfdf73eec50a3ea2b8e0f28ad89b507af2c0ce9f541126560a8: Status 404 returned error can't find the container with id 44443e303df2fcfdf73eec50a3ea2b8e0f28ad89b507af2c0ce9f541126560a8
Sep 16 12:54:40.957520 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:40.957493    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"
Sep 16 12:54:41.467284 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:41.467242    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7553-c9c26a" event={"ID":"697fa6b7-876f-4dbd-a03c-2913e6804adc","Type":"ContainerStarted","Data":"44443e303df2fcfdf73eec50a3ea2b8e0f28ad89b507af2c0ce9f541126560a8"}
Sep 16 12:54:41.467578 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:41.467552    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"
Sep 16 12:54:41.468563 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:41.468541    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"
Sep 16 12:54:43.480399 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:43.480308    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7553-c9c26a" event={"ID":"697fa6b7-876f-4dbd-a03c-2913e6804adc","Type":"ContainerStarted","Data":"4cec94dc44dcab5042786a1f245ca086b93c439a9a4faa5bde73817e41fe2279"}
Sep 16 12:54:43.496736 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:43.496682    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7553-c9c26a" podStartSLOduration=1.1632281899999999 podStartE2EDuration="3.49666593s" totalImagesPullingTime="2.33343774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:54:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:54:43.49487174 +0000 UTC m=+989.942104385" watchObservedRunningTime="2026-09-16 12:54:43.49666593 +0000 UTC m=+989.943898571"
Sep 16 12:54:44.088294 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:44.088262    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7553-c9c26a"]
Sep 16 12:54:44.094456 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:44.094417    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7553-c9c26a"]
Sep 16 12:54:44.485763 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:54:44.485733    2577 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7553-c9c26a" container="test-curl-7553-c9c26a"
Sep 16 12:54:45.390652 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:45.390620    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-490797-7c9447cf5d-gb8wg"]
Sep 16 12:54:45.391413 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:45.391385    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="697fa6b7-876f-4dbd-a03c-2913e6804adc" containerName="test-curl-7553-c9c26a"
Sep 16 12:54:45.391413 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:45.391410    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="697fa6b7-876f-4dbd-a03c-2913e6804adc" containerName="test-curl-7553-c9c26a"
Sep 16 12:54:45.391592 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:45.391499    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="697fa6b7-876f-4dbd-a03c-2913e6804adc" containerName="test-curl-7553-c9c26a"
Sep 16 12:54:45.394998 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:45.394978    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-490797-7c9447cf5d-gb8wg"
Sep 16 12:54:45.397909 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:45.397888    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-490797\""
Sep 16 12:54:45.397996 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:45.397885    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-490797-dockercfg-ltqcj\""
Sep 16 12:54:45.405605 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:45.405585    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-490797-7c9447cf5d-gb8wg"]
Sep 16 12:54:45.431667 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:45.431640    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb83e365-cc67-44bc-9a58-f1eb36ff6232-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-490797-7c9447cf5d-gb8wg\" (UID: \"bb83e365-cc67-44bc-9a58-f1eb36ff6232\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-490797-7c9447cf5d-gb8wg"
Sep 16 12:54:45.431832 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:45.431688    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbzr6\" (UniqueName: \"kubernetes.io/projected/bb83e365-cc67-44bc-9a58-f1eb36ff6232-kube-api-access-rbzr6\") pod \"payload-pre-processing-e2e-worker-w0-490797-7c9447cf5d-gb8wg\" (UID: \"bb83e365-cc67-44bc-9a58-f1eb36ff6232\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-490797-7c9447cf5d-gb8wg"
Sep 16 12:54:45.491955 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:45.491910    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7553-c9c26a" podUID="697fa6b7-876f-4dbd-a03c-2913e6804adc" containerName="test-curl-7553-c9c26a" containerID="cri-o://4cec94dc44dcab5042786a1f245ca086b93c439a9a4faa5bde73817e41fe2279" gracePeriod=2
Sep 16 12:54:45.496885 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:54:45.496857    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7553-c9c26a\" is forbidden: User \"system:node:ip-10-0-133-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-190.ec2.internal' and this object" podUID="697fa6b7-876f-4dbd-a03c-2913e6804adc" pod="opendatahub/test-curl-7553-c9c26a"
Sep 16 12:54:45.532474 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:45.532443    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbzr6\" (UniqueName: \"kubernetes.io/projected/bb83e365-cc67-44bc-9a58-f1eb36ff6232-kube-api-access-rbzr6\") pod \"payload-pre-processing-e2e-worker-w0-490797-7c9447cf5d-gb8wg\" (UID: \"bb83e365-cc67-44bc-9a58-f1eb36ff6232\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-490797-7c9447cf5d-gb8wg"
Sep 16 12:54:45.532590 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:45.532570    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb83e365-cc67-44bc-9a58-f1eb36ff6232-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-490797-7c9447cf5d-gb8wg\" (UID: \"bb83e365-cc67-44bc-9a58-f1eb36ff6232\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-490797-7c9447cf5d-gb8wg"
Sep 16 12:54:45.533363 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:45.533274    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb83e365-cc67-44bc-9a58-f1eb36ff6232-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-490797-7c9447cf5d-gb8wg\" (UID: \"bb83e365-cc67-44bc-9a58-f1eb36ff6232\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-490797-7c9447cf5d-gb8wg"
Sep 16 12:54:45.544391 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:45.544363    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbzr6\" (UniqueName: \"kubernetes.io/projected/bb83e365-cc67-44bc-9a58-f1eb36ff6232-kube-api-access-rbzr6\") pod \"payload-pre-processing-e2e-worker-w0-490797-7c9447cf5d-gb8wg\" (UID: \"bb83e365-cc67-44bc-9a58-f1eb36ff6232\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-490797-7c9447cf5d-gb8wg"
Sep 16 12:54:45.708469 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:45.708384    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-490797-7c9447cf5d-gb8wg"
Sep 16 12:54:45.987774 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:45.987747    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-490797-7c9447cf5d-gb8wg"]
Sep 16 12:54:45.989768 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:54:45.989736    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb83e365_cc67_44bc_9a58_f1eb36ff6232.slice/crio-f3bc9bc81062b10709f19e2a269e6a0a505b5a8099887f92e028b7cd2ac238ac WatchSource:0}: Error finding container f3bc9bc81062b10709f19e2a269e6a0a505b5a8099887f92e028b7cd2ac238ac: Status 404 returned error can't find the container with id f3bc9bc81062b10709f19e2a269e6a0a505b5a8099887f92e028b7cd2ac238ac
Sep 16 12:54:46.499159 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:46.499119    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-490797-7c9447cf5d-gb8wg" event={"ID":"bb83e365-cc67-44bc-9a58-f1eb36ff6232","Type":"ContainerStarted","Data":"f3bc9bc81062b10709f19e2a269e6a0a505b5a8099887f92e028b7cd2ac238ac"}
Sep 16 12:54:47.508809 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:47.508766    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-490797-7c9447cf5d-gb8wg" event={"ID":"bb83e365-cc67-44bc-9a58-f1eb36ff6232","Type":"ContainerStarted","Data":"4f1f15a6d57177724c50e840ca3c8791c652b9a3e8746bc53c3b880b14961d60"}
Sep 16 12:54:47.509376 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:47.509349    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-490797-7c9447cf5d-gb8wg"
Sep 16 12:54:47.532746 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:47.532691    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-490797-7c9447cf5d-gb8wg" podStartSLOduration=1.142081456 podStartE2EDuration="2.532673804s" totalImagesPullingTime="1.390592348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:54:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:54:47.530521755 +0000 UTC m=+993.977754424" watchObservedRunningTime="2026-09-16 12:54:47.532673804 +0000 UTC m=+993.979906445"
Sep 16 12:54:47.687520 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:47.687495    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7553-c9c26a"
Sep 16 12:54:47.751960 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:47.751930    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfzwn\" (UniqueName: \"kubernetes.io/projected/697fa6b7-876f-4dbd-a03c-2913e6804adc-kube-api-access-cfzwn\") pod \"697fa6b7-876f-4dbd-a03c-2913e6804adc\" (UID: \"697fa6b7-876f-4dbd-a03c-2913e6804adc\") "
Sep 16 12:54:47.754275 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:47.754250    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/697fa6b7-876f-4dbd-a03c-2913e6804adc-kube-api-access-cfzwn" (OuterVolumeSpecName: "kube-api-access-cfzwn") pod "697fa6b7-876f-4dbd-a03c-2913e6804adc" (UID: "697fa6b7-876f-4dbd-a03c-2913e6804adc"). InnerVolumeSpecName "kube-api-access-cfzwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:54:47.853598 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:47.853562    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfzwn\" (UniqueName: \"kubernetes.io/projected/697fa6b7-876f-4dbd-a03c-2913e6804adc-kube-api-access-cfzwn\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:54:48.054866 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:48.054832    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="697fa6b7-876f-4dbd-a03c-2913e6804adc" path="/var/lib/kubelet/pods/697fa6b7-876f-4dbd-a03c-2913e6804adc/volumes"
Sep 16 12:54:48.515562 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:48.515534    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"]
Sep 16 12:54:48.516464 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:48.516442    2577 generic.go:356] "Generic (PLEG): container finished" podID="697fa6b7-876f-4dbd-a03c-2913e6804adc" containerID="4cec94dc44dcab5042786a1f245ca086b93c439a9a4faa5bde73817e41fe2279" exitCode=137
Sep 16 12:54:48.516570 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:48.516554    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7553-c9c26a"
Sep 16 12:54:48.519422 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:48.519404    2577 scope.go:117] "RemoveContainer" containerID="4cec94dc44dcab5042786a1f245ca086b93c439a9a4faa5bde73817e41fe2279"
Sep 16 12:54:48.519529 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:48.519512    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"
Sep 16 12:54:48.525203 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:48.525185    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-79dbf9e9-openshift-default-dockercfg-hpls9\""
Sep 16 12:54:48.529223 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:48.529207    2577 scope.go:117] "RemoveContainer" containerID="4cec94dc44dcab5042786a1f245ca086b93c439a9a4faa5bde73817e41fe2279"
Sep 16 12:54:48.529515 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:54:48.529498    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cec94dc44dcab5042786a1f245ca086b93c439a9a4faa5bde73817e41fe2279\": container with ID starting with 4cec94dc44dcab5042786a1f245ca086b93c439a9a4faa5bde73817e41fe2279 not found: ID does not exist" containerID="4cec94dc44dcab5042786a1f245ca086b93c439a9a4faa5bde73817e41fe2279"
Sep 16 12:54:48.529582 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:48.529526    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cec94dc44dcab5042786a1f245ca086b93c439a9a4faa5bde73817e41fe2279"} err="failed to get container status \"4cec94dc44dcab5042786a1f245ca086b93c439a9a4faa5bde73817e41fe2279\": rpc error: code = NotFound desc = could not find container \"4cec94dc44dcab5042786a1f245ca086b93c439a9a4faa5bde73817e41fe2279\": container with ID starting with 4cec94dc44dcab5042786a1f245ca086b93c439a9a4faa5bde73817e41fe2279 not found: ID does not exist"
Sep 16 12:54:48.541355 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:48.541331    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"]
Sep 16 12:54:48.558764 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:48.558736    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a5c65e-485d-480a-9cb4-2e306d451074-credential-socket\") pod \"e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j\" (UID: \"b9a5c65e-485d-480a-9cb4-2e306d451074\") " pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"
Sep 16 12:54:48.558855 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:48.558798    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a5c65e-485d-480a-9cb4-2e306d451074-workload-socket\") pod \"e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j\" (UID: \"b9a5c65e-485d-480a-9cb4-2e306d451074\") " pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"
Sep 16 12:54:48.558855 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:48.558849    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9a5c65e-485d-480a-9cb4-2e306d451074-istio-envoy\") pod \"e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j\" (UID: \"b9a5c65e-485d-480a-9cb4-2e306d451074\") " pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"
Sep 16 12:54:48.558954 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:48.558899    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9a5c65e-485d-480a-9cb4-2e306d451074-istio-podinfo\") pod \"e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j\" (UID: \"b9a5c65e-485d-480a-9cb4-2e306d451074\") " pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"
Sep 16 12:54:48.558954 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:48.558927    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9a5c65e-485d-480a-9cb4-2e306d451074-istiod-ca-cert\") pod \"e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j\" (UID: \"b9a5c65e-485d-480a-9cb4-2e306d451074\") " pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"
Sep 16 12:54:48.559062 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:48.558989    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxwck\" (UniqueName: \"kubernetes.io/projected/b9a5c65e-485d-480a-9cb4-2e306d451074-kube-api-access-pxwck\") pod \"e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j\" (UID: \"b9a5c65e-485d-480a-9cb4-2e306d451074\") " pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"
Sep 16 12:54:48.559062 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:48.559045    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9a5c65e-485d-480a-9cb4-2e306d451074-workload-certs\") pod \"e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j\" (UID: \"b9a5c65e-485d-480a-9cb4-2e306d451074\") " pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"
Sep 16 12:54:48.559160 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:48.559134    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9a5c65e-485d-480a-9cb4-2e306d451074-istio-token\") pod \"e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j\" (UID: \"b9a5c65e-485d-480a-9cb4-2e306d451074\") " pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"
Sep 16 12:54:48.559221 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:48.559205    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9a5c65e-485d-480a-9cb4-2e306d451074-istio-data\") pod \"e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j\" (UID: \"b9a5c65e-485d-480a-9cb4-2e306d451074\") " pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"
Sep 16 12:54:48.660707 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:48.660675    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a5c65e-485d-480a-9cb4-2e306d451074-workload-socket\") pod \"e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j\" (UID: \"b9a5c65e-485d-480a-9cb4-2e306d451074\") " pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"
Sep 16 12:54:48.660707 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:48.660714    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9a5c65e-485d-480a-9cb4-2e306d451074-istio-envoy\") pod \"e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j\" (UID: \"b9a5c65e-485d-480a-9cb4-2e306d451074\") " pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"
Sep 16 12:54:48.660954 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:48.660755    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9a5c65e-485d-480a-9cb4-2e306d451074-istio-podinfo\") pod \"e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j\" (UID: \"b9a5c65e-485d-480a-9cb4-2e306d451074\") " pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"
Sep 16 12:54:48.660954 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:48.660776    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9a5c65e-485d-480a-9cb4-2e306d451074-istiod-ca-cert\") pod \"e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j\" (UID: \"b9a5c65e-485d-480a-9cb4-2e306d451074\") " pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"
Sep 16 12:54:48.660954 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:48.660803    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxwck\" (UniqueName: \"kubernetes.io/projected/b9a5c65e-485d-480a-9cb4-2e306d451074-kube-api-access-pxwck\") pod \"e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j\" (UID: \"b9a5c65e-485d-480a-9cb4-2e306d451074\") " pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"
Sep 16 12:54:48.660954 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:48.660825    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9a5c65e-485d-480a-9cb4-2e306d451074-workload-certs\") pod \"e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j\" (UID: \"b9a5c65e-485d-480a-9cb4-2e306d451074\") " pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"
Sep 16 12:54:48.660954 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:48.660939    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9a5c65e-485d-480a-9cb4-2e306d451074-istio-token\") pod \"e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j\" (UID: \"b9a5c65e-485d-480a-9cb4-2e306d451074\") " pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"
Sep 16 12:54:48.661231 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:48.660995    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9a5c65e-485d-480a-9cb4-2e306d451074-istio-data\") pod \"e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j\" (UID: \"b9a5c65e-485d-480a-9cb4-2e306d451074\") " pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"
Sep 16 12:54:48.661231 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:48.661069    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a5c65e-485d-480a-9cb4-2e306d451074-credential-socket\") pod \"e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j\" (UID: \"b9a5c65e-485d-480a-9cb4-2e306d451074\") " pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"
Sep 16 12:54:48.661231 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:48.661166    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a5c65e-485d-480a-9cb4-2e306d451074-workload-socket\") pod \"e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j\" (UID: \"b9a5c65e-485d-480a-9cb4-2e306d451074\") " pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"
Sep 16 12:54:48.661376 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:48.661249    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9a5c65e-485d-480a-9cb4-2e306d451074-workload-certs\") pod \"e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j\" (UID: \"b9a5c65e-485d-480a-9cb4-2e306d451074\") " pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"
Sep 16 12:54:48.661412 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:48.661389    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a5c65e-485d-480a-9cb4-2e306d451074-credential-socket\") pod \"e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j\" (UID: \"b9a5c65e-485d-480a-9cb4-2e306d451074\") " pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"
Sep 16 12:54:48.661484 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:48.661464    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9a5c65e-485d-480a-9cb4-2e306d451074-istio-data\") pod \"e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j\" (UID: \"b9a5c65e-485d-480a-9cb4-2e306d451074\") " pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"
Sep 16 12:54:48.661604 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:48.661546    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9a5c65e-485d-480a-9cb4-2e306d451074-istiod-ca-cert\") pod \"e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j\" (UID: \"b9a5c65e-485d-480a-9cb4-2e306d451074\") " pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"
Sep 16 12:54:48.663249 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:48.663228    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9a5c65e-485d-480a-9cb4-2e306d451074-istio-envoy\") pod \"e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j\" (UID: \"b9a5c65e-485d-480a-9cb4-2e306d451074\") " pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"
Sep 16 12:54:48.663599 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:48.663583    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9a5c65e-485d-480a-9cb4-2e306d451074-istio-podinfo\") pod \"e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j\" (UID: \"b9a5c65e-485d-480a-9cb4-2e306d451074\") " pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"
Sep 16 12:54:48.679993 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:48.679966    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9a5c65e-485d-480a-9cb4-2e306d451074-istio-token\") pod \"e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j\" (UID: \"b9a5c65e-485d-480a-9cb4-2e306d451074\") " pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"
Sep 16 12:54:48.682674 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:48.682654    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxwck\" (UniqueName: \"kubernetes.io/projected/b9a5c65e-485d-480a-9cb4-2e306d451074-kube-api-access-pxwck\") pod \"e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j\" (UID: \"b9a5c65e-485d-480a-9cb4-2e306d451074\") " pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"
Sep 16 12:54:48.830568 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:48.830483    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"
Sep 16 12:54:49.113367 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.113335    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"]
Sep 16 12:54:49.119187 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.119136    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:54:49.119439 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.119414    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:54:49.119657 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.119633    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:54:49.170636 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.170609    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"]
Sep 16 12:54:49.175763 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.175744    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"
Sep 16 12:54:49.178643 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.178623    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-a24d8b-openshift-default-dockercfg-6x2cn\""
Sep 16 12:54:49.187967 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.187941    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"]
Sep 16 12:54:49.269762 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.269217    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1adf6c30-6569-40a5-ba22-caf26b38f80a-istio-podinfo\") pod \"e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm\" (UID: \"1adf6c30-6569-40a5-ba22-caf26b38f80a\") " pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"
Sep 16 12:54:49.269762 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.269294    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1adf6c30-6569-40a5-ba22-caf26b38f80a-istio-data\") pod \"e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm\" (UID: \"1adf6c30-6569-40a5-ba22-caf26b38f80a\") " pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"
Sep 16 12:54:49.269762 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.269345    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1adf6c30-6569-40a5-ba22-caf26b38f80a-istio-envoy\") pod \"e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm\" (UID: \"1adf6c30-6569-40a5-ba22-caf26b38f80a\") " pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"
Sep 16 12:54:49.269762 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.269387    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adf6c30-6569-40a5-ba22-caf26b38f80a-credential-socket\") pod \"e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm\" (UID: \"1adf6c30-6569-40a5-ba22-caf26b38f80a\") " pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"
Sep 16 12:54:49.269762 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.269419    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adf6c30-6569-40a5-ba22-caf26b38f80a-workload-socket\") pod \"e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm\" (UID: \"1adf6c30-6569-40a5-ba22-caf26b38f80a\") " pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"
Sep 16 12:54:49.269762 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.269468    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1adf6c30-6569-40a5-ba22-caf26b38f80a-workload-certs\") pod \"e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm\" (UID: \"1adf6c30-6569-40a5-ba22-caf26b38f80a\") " pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"
Sep 16 12:54:49.269762 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.269523    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1adf6c30-6569-40a5-ba22-caf26b38f80a-istiod-ca-cert\") pod \"e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm\" (UID: \"1adf6c30-6569-40a5-ba22-caf26b38f80a\") " pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"
Sep 16 12:54:49.269762 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.269584    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77s89\" (UniqueName: \"kubernetes.io/projected/1adf6c30-6569-40a5-ba22-caf26b38f80a-kube-api-access-77s89\") pod \"e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm\" (UID: \"1adf6c30-6569-40a5-ba22-caf26b38f80a\") " pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"
Sep 16 12:54:49.269762 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.269615    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1adf6c30-6569-40a5-ba22-caf26b38f80a-istio-token\") pod \"e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm\" (UID: \"1adf6c30-6569-40a5-ba22-caf26b38f80a\") " pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"
Sep 16 12:54:49.370260 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.370170    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1adf6c30-6569-40a5-ba22-caf26b38f80a-istio-data\") pod \"e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm\" (UID: \"1adf6c30-6569-40a5-ba22-caf26b38f80a\") " pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"
Sep 16 12:54:49.370260 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.370209    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1adf6c30-6569-40a5-ba22-caf26b38f80a-istio-envoy\") pod \"e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm\" (UID: \"1adf6c30-6569-40a5-ba22-caf26b38f80a\") " pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"
Sep 16 12:54:49.370260 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.370238    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adf6c30-6569-40a5-ba22-caf26b38f80a-credential-socket\") pod \"e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm\" (UID: \"1adf6c30-6569-40a5-ba22-caf26b38f80a\") " pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"
Sep 16 12:54:49.370502 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.370264    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adf6c30-6569-40a5-ba22-caf26b38f80a-workload-socket\") pod \"e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm\" (UID: \"1adf6c30-6569-40a5-ba22-caf26b38f80a\") " pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"
Sep 16 12:54:49.370502 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.370315    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1adf6c30-6569-40a5-ba22-caf26b38f80a-workload-certs\") pod \"e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm\" (UID: \"1adf6c30-6569-40a5-ba22-caf26b38f80a\") " pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"
Sep 16 12:54:49.370502 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.370361    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1adf6c30-6569-40a5-ba22-caf26b38f80a-istiod-ca-cert\") pod \"e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm\" (UID: \"1adf6c30-6569-40a5-ba22-caf26b38f80a\") " pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"
Sep 16 12:54:49.370502 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.370407    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77s89\" (UniqueName: \"kubernetes.io/projected/1adf6c30-6569-40a5-ba22-caf26b38f80a-kube-api-access-77s89\") pod \"e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm\" (UID: \"1adf6c30-6569-40a5-ba22-caf26b38f80a\") " pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"
Sep 16 12:54:49.370502 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.370434    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1adf6c30-6569-40a5-ba22-caf26b38f80a-istio-token\") pod \"e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm\" (UID: \"1adf6c30-6569-40a5-ba22-caf26b38f80a\") " pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"
Sep 16 12:54:49.370502 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.370488    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1adf6c30-6569-40a5-ba22-caf26b38f80a-istio-podinfo\") pod \"e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm\" (UID: \"1adf6c30-6569-40a5-ba22-caf26b38f80a\") " pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"
Sep 16 12:54:49.370746 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.370687    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1adf6c30-6569-40a5-ba22-caf26b38f80a-istio-data\") pod \"e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm\" (UID: \"1adf6c30-6569-40a5-ba22-caf26b38f80a\") " pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"
Sep 16 12:54:49.370808 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.370779    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adf6c30-6569-40a5-ba22-caf26b38f80a-credential-socket\") pod \"e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm\" (UID: \"1adf6c30-6569-40a5-ba22-caf26b38f80a\") " pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"
Sep 16 12:54:49.371198 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.371110    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adf6c30-6569-40a5-ba22-caf26b38f80a-workload-socket\") pod \"e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm\" (UID: \"1adf6c30-6569-40a5-ba22-caf26b38f80a\") " pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"
Sep 16 12:54:49.371341 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.371199    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1adf6c30-6569-40a5-ba22-caf26b38f80a-workload-certs\") pod \"e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm\" (UID: \"1adf6c30-6569-40a5-ba22-caf26b38f80a\") " pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"
Sep 16 12:54:49.371489 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.371451    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1adf6c30-6569-40a5-ba22-caf26b38f80a-istiod-ca-cert\") pod \"e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm\" (UID: \"1adf6c30-6569-40a5-ba22-caf26b38f80a\") " pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"
Sep 16 12:54:49.373107 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.373085    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1adf6c30-6569-40a5-ba22-caf26b38f80a-istio-podinfo\") pod \"e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm\" (UID: \"1adf6c30-6569-40a5-ba22-caf26b38f80a\") " pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"
Sep 16 12:54:49.373461 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.373446    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1adf6c30-6569-40a5-ba22-caf26b38f80a-istio-envoy\") pod \"e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm\" (UID: \"1adf6c30-6569-40a5-ba22-caf26b38f80a\") " pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"
Sep 16 12:54:49.378944 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.378923    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1adf6c30-6569-40a5-ba22-caf26b38f80a-istio-token\") pod \"e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm\" (UID: \"1adf6c30-6569-40a5-ba22-caf26b38f80a\") " pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"
Sep 16 12:54:49.379230 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.379212    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77s89\" (UniqueName: \"kubernetes.io/projected/1adf6c30-6569-40a5-ba22-caf26b38f80a-kube-api-access-77s89\") pod \"e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm\" (UID: \"1adf6c30-6569-40a5-ba22-caf26b38f80a\") " pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"
Sep 16 12:54:49.487097 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.487060    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"
Sep 16 12:54:49.525198 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.525152    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j" event={"ID":"b9a5c65e-485d-480a-9cb4-2e306d451074","Type":"ContainerStarted","Data":"37f90da39d54be6cf90bc5e35155aeaf16f97f8534096e50d2407265f40af0b1"}
Sep 16 12:54:49.525601 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.525258    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j" event={"ID":"b9a5c65e-485d-480a-9cb4-2e306d451074","Type":"ContainerStarted","Data":"05672875a1bd265b888103edbaad7fd5a4de446e59b19b8a99c6c8f9d15dec56"}
Sep 16 12:54:49.565895 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.565785    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j" podStartSLOduration=1.565765411 podStartE2EDuration="1.565765411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:54:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:54:49.565453156 +0000 UTC m=+996.012685825" watchObservedRunningTime="2026-09-16 12:54:49.565765411 +0000 UTC m=+996.012998055"
Sep 16 12:54:49.771692 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.771665    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"]
Sep 16 12:54:49.775846 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:54:49.775793    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1adf6c30_6569_40a5_ba22_caf26b38f80a.slice/crio-9ecfcdb3535f387764a89a741c85ecd419ec4459a40baf0fcb55faa433a7436a WatchSource:0}: Error finding container 9ecfcdb3535f387764a89a741c85ecd419ec4459a40baf0fcb55faa433a7436a: Status 404 returned error can't find the container with id 9ecfcdb3535f387764a89a741c85ecd419ec4459a40baf0fcb55faa433a7436a
Sep 16 12:54:49.778804 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.778769    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:54:49.778898 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.778856    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:54:49.778963 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.778898    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:54:49.831016 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:49.830987    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"
Sep 16 12:54:50.532350 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:50.532310    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm" event={"ID":"1adf6c30-6569-40a5-ba22-caf26b38f80a","Type":"ContainerStarted","Data":"d960f8e232219bba39dd1c3f1886c89cff0d829e78db30c74539978576ab7aaf"}
Sep 16 12:54:50.532843 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:50.532380    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm" event={"ID":"1adf6c30-6569-40a5-ba22-caf26b38f80a","Type":"ContainerStarted","Data":"9ecfcdb3535f387764a89a741c85ecd419ec4459a40baf0fcb55faa433a7436a"}
Sep 16 12:54:50.555150 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:50.555102    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm" podStartSLOduration=1.55508106 podStartE2EDuration="1.55508106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:54:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:54:50.55381278 +0000 UTC m=+997.001045422" watchObservedRunningTime="2026-09-16 12:54:50.55508106 +0000 UTC m=+997.002313702"
Sep 16 12:54:50.836718 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:50.836637    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"
Sep 16 12:54:51.487832 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:51.487507    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"
Sep 16 12:54:51.512006 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:51.511978    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"
Sep 16 12:54:51.549055 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:51.548152    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"
Sep 16 12:54:51.549055 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:51.548208    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"
Sep 16 12:54:51.550541 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:51.550321    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"
Sep 16 12:54:51.557289 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:51.557270    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"
Sep 16 12:54:57.509240 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:57.509196    2577 patch_prober.go:28] interesting pod/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 12:54:57.509796 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:57.509263    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz" podUID="99b70637-81ce-4503-bc3d-de5dc69e7b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 12:54:57.795208 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:57.795121    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-79dbf9e9-7d59c45f4-j5mv4"]
Sep 16 12:54:57.800603 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:57.800580    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-79dbf9e9-7d59c45f4-j5mv4"
Sep 16 12:54:57.805163 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:57.805134    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-79dbf9e9\""
Sep 16 12:54:57.805826 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:57.805801    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-79dbf9e9-dockercfg-t4wpp\""
Sep 16 12:54:57.831287 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:57.831206    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-79dbf9e9-7d59c45f4-j5mv4"]
Sep 16 12:54:57.895472 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:57.895437    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7d8ce4-7b7f-4df0-b89f-24f52d065495-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-79dbf9e9-7d59c45f4-j5mv4\" (UID: \"9a7d8ce4-7b7f-4df0-b89f-24f52d065495\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-79dbf9e9-7d59c45f4-j5mv4"
Sep 16 12:54:57.895707 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:57.895491    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d44qq\" (UniqueName: \"kubernetes.io/projected/9a7d8ce4-7b7f-4df0-b89f-24f52d065495-kube-api-access-d44qq\") pod \"payload-pre-processing-e2e-ait-79dbf9e9-7d59c45f4-j5mv4\" (UID: \"9a7d8ce4-7b7f-4df0-b89f-24f52d065495\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-79dbf9e9-7d59c45f4-j5mv4"
Sep 16 12:54:57.996648 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:57.996608    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7d8ce4-7b7f-4df0-b89f-24f52d065495-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-79dbf9e9-7d59c45f4-j5mv4\" (UID: \"9a7d8ce4-7b7f-4df0-b89f-24f52d065495\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-79dbf9e9-7d59c45f4-j5mv4"
Sep 16 12:54:57.996854 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:57.996675    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d44qq\" (UniqueName: \"kubernetes.io/projected/9a7d8ce4-7b7f-4df0-b89f-24f52d065495-kube-api-access-d44qq\") pod \"payload-pre-processing-e2e-ait-79dbf9e9-7d59c45f4-j5mv4\" (UID: \"9a7d8ce4-7b7f-4df0-b89f-24f52d065495\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-79dbf9e9-7d59c45f4-j5mv4"
Sep 16 12:54:57.997435 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:57.997414    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7d8ce4-7b7f-4df0-b89f-24f52d065495-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-79dbf9e9-7d59c45f4-j5mv4\" (UID: \"9a7d8ce4-7b7f-4df0-b89f-24f52d065495\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-79dbf9e9-7d59c45f4-j5mv4"
Sep 16 12:54:58.006449 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:58.006420    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d44qq\" (UniqueName: \"kubernetes.io/projected/9a7d8ce4-7b7f-4df0-b89f-24f52d065495-kube-api-access-d44qq\") pod \"payload-pre-processing-e2e-ait-79dbf9e9-7d59c45f4-j5mv4\" (UID: \"9a7d8ce4-7b7f-4df0-b89f-24f52d065495\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-79dbf9e9-7d59c45f4-j5mv4"
Sep 16 12:54:58.114792 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:58.114699    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-79dbf9e9-7d59c45f4-j5mv4"
Sep 16 12:54:58.415803 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:58.415767    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-79dbf9e9-7d59c45f4-j5mv4"]
Sep 16 12:54:58.416487 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:54:58.416455    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a7d8ce4_7b7f_4df0_b89f_24f52d065495.slice/crio-1c73ccec8e1567f4d039bb2293d5f7b61542612d2ce91e0073d3e3d4c8348b58 WatchSource:0}: Error finding container 1c73ccec8e1567f4d039bb2293d5f7b61542612d2ce91e0073d3e3d4c8348b58: Status 404 returned error can't find the container with id 1c73ccec8e1567f4d039bb2293d5f7b61542612d2ce91e0073d3e3d4c8348b58
Sep 16 12:54:58.519659 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:58.519627    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-490797-7c9447cf5d-gb8wg"
Sep 16 12:54:58.597717 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:58.597676    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-79dbf9e9-7d59c45f4-j5mv4" event={"ID":"9a7d8ce4-7b7f-4df0-b89f-24f52d065495","Type":"ContainerStarted","Data":"4118ebc57649ba1d249e612f29253ffc56ee0c4e69e1d7d0c10c285aa597dc82"}
Sep 16 12:54:58.597901 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:58.597741    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-79dbf9e9-7d59c45f4-j5mv4" event={"ID":"9a7d8ce4-7b7f-4df0-b89f-24f52d065495","Type":"ContainerStarted","Data":"1c73ccec8e1567f4d039bb2293d5f7b61542612d2ce91e0073d3e3d4c8348b58"}
Sep 16 12:54:58.597901 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:58.597800    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-79dbf9e9-7d59c45f4-j5mv4"
Sep 16 12:54:58.620793 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:54:58.620732    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-79dbf9e9-7d59c45f4-j5mv4" podStartSLOduration=1.620712348 podStartE2EDuration="1.620712348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:54:58.617934336 +0000 UTC m=+1005.065166979" watchObservedRunningTime="2026-09-16 12:54:58.620712348 +0000 UTC m=+1005.067944993"
Sep 16 12:55:00.613439 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.613401    2577 generic.go:356] "Generic (PLEG): container finished" podID="99b70637-81ce-4503-bc3d-de5dc69e7b97" containerID="472837c031edae40d3d5c05ad30b7ed63e00da96d92d905048f17581c59547d8" exitCode=0
Sep 16 12:55:00.613974 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.613481    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz" event={"ID":"99b70637-81ce-4503-bc3d-de5dc69e7b97","Type":"ContainerDied","Data":"472837c031edae40d3d5c05ad30b7ed63e00da96d92d905048f17581c59547d8"}
Sep 16 12:55:00.614087 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.614068    2577 scope.go:117] "RemoveContainer" containerID="472837c031edae40d3d5c05ad30b7ed63e00da96d92d905048f17581c59547d8"
Sep 16 12:55:00.616058 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.616002    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:55:00.616183 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.616107    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:55:00.616183 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.616154    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:55:00.616306 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.616241    2577 generic.go:356] "Generic (PLEG): container finished" podID="462e0727-223b-4217-af33-aae07198ca07" containerID="c8f872f61ddab2e36fd291728ca8cf8f7faa954725d521c6dc6c328eb7775045" exitCode=0
Sep 16 12:55:00.616365 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.616336    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs" event={"ID":"462e0727-223b-4217-af33-aae07198ca07","Type":"ContainerDied","Data":"c8f872f61ddab2e36fd291728ca8cf8f7faa954725d521c6dc6c328eb7775045"}
Sep 16 12:55:00.616798 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.616780    2577 scope.go:117] "RemoveContainer" containerID="c8f872f61ddab2e36fd291728ca8cf8f7faa954725d521c6dc6c328eb7775045"
Sep 16 12:55:00.618077 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.618009    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:55:00.618191 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.618122    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:55:00.618191 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.618172    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:55:00.628966 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.628471    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"]
Sep 16 12:55:00.633773 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.633755    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"
Sep 16 12:55:00.639903 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.639882    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-89d6bf-openshift-default-dockercfg-wpfft\""
Sep 16 12:55:00.649138 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.649111    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"]
Sep 16 12:55:00.729985 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.729366    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10ebb745-2b51-4ac4-8834-66601208abdb-workload-certs\") pod \"e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888\" (UID: \"10ebb745-2b51-4ac4-8834-66601208abdb\") " pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"
Sep 16 12:55:00.729985 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.729484    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10ebb745-2b51-4ac4-8834-66601208abdb-istio-data\") pod \"e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888\" (UID: \"10ebb745-2b51-4ac4-8834-66601208abdb\") " pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"
Sep 16 12:55:00.729985 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.729526    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ebb745-2b51-4ac4-8834-66601208abdb-credential-socket\") pod \"e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888\" (UID: \"10ebb745-2b51-4ac4-8834-66601208abdb\") " pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"
Sep 16 12:55:00.729985 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.729606    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ebb745-2b51-4ac4-8834-66601208abdb-workload-socket\") pod \"e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888\" (UID: \"10ebb745-2b51-4ac4-8834-66601208abdb\") " pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"
Sep 16 12:55:00.729985 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.729648    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10ebb745-2b51-4ac4-8834-66601208abdb-istio-podinfo\") pod \"e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888\" (UID: \"10ebb745-2b51-4ac4-8834-66601208abdb\") " pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"
Sep 16 12:55:00.729985 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.729672    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10ebb745-2b51-4ac4-8834-66601208abdb-istio-token\") pod \"e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888\" (UID: \"10ebb745-2b51-4ac4-8834-66601208abdb\") " pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"
Sep 16 12:55:00.729985 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.729767    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10ebb745-2b51-4ac4-8834-66601208abdb-istiod-ca-cert\") pod \"e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888\" (UID: \"10ebb745-2b51-4ac4-8834-66601208abdb\") " pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"
Sep 16 12:55:00.729985 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.729805    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7jxb\" (UniqueName: \"kubernetes.io/projected/10ebb745-2b51-4ac4-8834-66601208abdb-kube-api-access-g7jxb\") pod \"e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888\" (UID: \"10ebb745-2b51-4ac4-8834-66601208abdb\") " pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"
Sep 16 12:55:00.730541 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.730190    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10ebb745-2b51-4ac4-8834-66601208abdb-istio-envoy\") pod \"e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888\" (UID: \"10ebb745-2b51-4ac4-8834-66601208abdb\") " pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"
Sep 16 12:55:00.831681 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.831646    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10ebb745-2b51-4ac4-8834-66601208abdb-workload-certs\") pod \"e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888\" (UID: \"10ebb745-2b51-4ac4-8834-66601208abdb\") " pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"
Sep 16 12:55:00.831848 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.831708    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10ebb745-2b51-4ac4-8834-66601208abdb-istio-data\") pod \"e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888\" (UID: \"10ebb745-2b51-4ac4-8834-66601208abdb\") " pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"
Sep 16 12:55:00.831848 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.831736    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ebb745-2b51-4ac4-8834-66601208abdb-credential-socket\") pod \"e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888\" (UID: \"10ebb745-2b51-4ac4-8834-66601208abdb\") " pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"
Sep 16 12:55:00.831848 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.831760    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ebb745-2b51-4ac4-8834-66601208abdb-workload-socket\") pod \"e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888\" (UID: \"10ebb745-2b51-4ac4-8834-66601208abdb\") " pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"
Sep 16 12:55:00.831848 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.831778    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10ebb745-2b51-4ac4-8834-66601208abdb-istio-podinfo\") pod \"e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888\" (UID: \"10ebb745-2b51-4ac4-8834-66601208abdb\") " pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"
Sep 16 12:55:00.831848 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.831795    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10ebb745-2b51-4ac4-8834-66601208abdb-istio-token\") pod \"e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888\" (UID: \"10ebb745-2b51-4ac4-8834-66601208abdb\") " pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"
Sep 16 12:55:00.831848 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.831835    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10ebb745-2b51-4ac4-8834-66601208abdb-istiod-ca-cert\") pod \"e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888\" (UID: \"10ebb745-2b51-4ac4-8834-66601208abdb\") " pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"
Sep 16 12:55:00.831848 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.831852    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7jxb\" (UniqueName: \"kubernetes.io/projected/10ebb745-2b51-4ac4-8834-66601208abdb-kube-api-access-g7jxb\") pod \"e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888\" (UID: \"10ebb745-2b51-4ac4-8834-66601208abdb\") " pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"
Sep 16 12:55:00.832159 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.831891    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10ebb745-2b51-4ac4-8834-66601208abdb-istio-envoy\") pod \"e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888\" (UID: \"10ebb745-2b51-4ac4-8834-66601208abdb\") " pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"
Sep 16 12:55:00.832529 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.832502    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ebb745-2b51-4ac4-8834-66601208abdb-workload-socket\") pod \"e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888\" (UID: \"10ebb745-2b51-4ac4-8834-66601208abdb\") " pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"
Sep 16 12:55:00.832921 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.832665    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10ebb745-2b51-4ac4-8834-66601208abdb-workload-certs\") pod \"e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888\" (UID: \"10ebb745-2b51-4ac4-8834-66601208abdb\") " pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"
Sep 16 12:55:00.832921 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.832860    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10ebb745-2b51-4ac4-8834-66601208abdb-istio-data\") pod \"e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888\" (UID: \"10ebb745-2b51-4ac4-8834-66601208abdb\") " pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"
Sep 16 12:55:00.833317 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.833206    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10ebb745-2b51-4ac4-8834-66601208abdb-istiod-ca-cert\") pod \"e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888\" (UID: \"10ebb745-2b51-4ac4-8834-66601208abdb\") " pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"
Sep 16 12:55:00.833861 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.833670    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ebb745-2b51-4ac4-8834-66601208abdb-credential-socket\") pod \"e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888\" (UID: \"10ebb745-2b51-4ac4-8834-66601208abdb\") " pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"
Sep 16 12:55:00.835051 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.834877    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10ebb745-2b51-4ac4-8834-66601208abdb-istio-envoy\") pod \"e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888\" (UID: \"10ebb745-2b51-4ac4-8834-66601208abdb\") " pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"
Sep 16 12:55:00.836220 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.836193    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10ebb745-2b51-4ac4-8834-66601208abdb-istio-podinfo\") pod \"e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888\" (UID: \"10ebb745-2b51-4ac4-8834-66601208abdb\") " pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"
Sep 16 12:55:00.845004 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.844977    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10ebb745-2b51-4ac4-8834-66601208abdb-istio-token\") pod \"e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888\" (UID: \"10ebb745-2b51-4ac4-8834-66601208abdb\") " pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"
Sep 16 12:55:00.845492 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.845471    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7jxb\" (UniqueName: \"kubernetes.io/projected/10ebb745-2b51-4ac4-8834-66601208abdb-kube-api-access-g7jxb\") pod \"e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888\" (UID: \"10ebb745-2b51-4ac4-8834-66601208abdb\") " pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"
Sep 16 12:55:00.986360 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:00.986272    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"
Sep 16 12:55:01.293716 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:01.293683    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"]
Sep 16 12:55:01.296646 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:55:01.296619    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10ebb745_2b51_4ac4_8834_66601208abdb.slice/crio-7455e21e67f53d12abcdb21f226545106333679041f3d5a5549855edc079b6ef WatchSource:0}: Error finding container 7455e21e67f53d12abcdb21f226545106333679041f3d5a5549855edc079b6ef: Status 404 returned error can't find the container with id 7455e21e67f53d12abcdb21f226545106333679041f3d5a5549855edc079b6ef
Sep 16 12:55:01.298898 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:01.298870    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:55:01.298964 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:01.298928    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:55:01.298964 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:01.298958    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:55:01.625492 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:01.625405    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz" event={"ID":"99b70637-81ce-4503-bc3d-de5dc69e7b97","Type":"ContainerStarted","Data":"dc0cf4a0f8fa83db54422e662098bda0c4142e1a1339200e0e025e018eae94f7"}
Sep 16 12:55:01.627296 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:01.627260    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888" event={"ID":"10ebb745-2b51-4ac4-8834-66601208abdb","Type":"ContainerStarted","Data":"15d28b872da26825994a5e63c24434579b6629a7aa70ab960c87994de2843d68"}
Sep 16 12:55:01.627443 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:01.627314    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888" event={"ID":"10ebb745-2b51-4ac4-8834-66601208abdb","Type":"ContainerStarted","Data":"7455e21e67f53d12abcdb21f226545106333679041f3d5a5549855edc079b6ef"}
Sep 16 12:55:01.629289 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:01.629264    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs" event={"ID":"462e0727-223b-4217-af33-aae07198ca07","Type":"ContainerStarted","Data":"72aa2e5afdc2a11695fe68dc2b8000ba423faccc22453e7adc282ff543039066"}
Sep 16 12:55:01.693727 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:01.693659    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888" podStartSLOduration=1.693636604 podStartE2EDuration="1.693636604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:55:01.693006679 +0000 UTC m=+1008.140239322" watchObservedRunningTime="2026-09-16 12:55:01.693636604 +0000 UTC m=+1008.140869248"
Sep 16 12:55:01.951807 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:01.951714    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"
Sep 16 12:55:01.951992 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:01.951947    2577 patch_prober.go:28] interesting pod/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 16 12:55:01.952109 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:01.952013    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz" podUID="99b70637-81ce-4503-bc3d-de5dc69e7b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 16 12:55:01.987150 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:01.987120    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"
Sep 16 12:55:02.026101 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:02.026068    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"
Sep 16 12:55:02.026337 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:02.026315    2577 patch_prober.go:28] interesting pod/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Sep 16 12:55:02.026391 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:02.026369    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs" podUID="462e0727-223b-4217-af33-aae07198ca07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 16 12:55:02.951961 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:02.951921    2577 patch_prober.go:28] interesting pod/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 16 12:55:02.952358 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:02.951983    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz" podUID="99b70637-81ce-4503-bc3d-de5dc69e7b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 16 12:55:02.996607 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:02.996567    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"
Sep 16 12:55:03.026373 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:03.026334    2577 patch_prober.go:28] interesting pod/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Sep 16 12:55:03.026665 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:03.026635    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs" podUID="462e0727-223b-4217-af33-aae07198ca07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 16 12:55:03.646490 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:03.646459    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"
Sep 16 12:55:03.647467 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:03.647445    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"
Sep 16 12:55:03.952115 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:03.951995    2577 patch_prober.go:28] interesting pod/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 16 12:55:03.952115 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:03.952091    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz" podUID="99b70637-81ce-4503-bc3d-de5dc69e7b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 16 12:55:04.026120 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:04.026080    2577 patch_prober.go:28] interesting pod/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Sep 16 12:55:04.026406 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:04.026373    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs" podUID="462e0727-223b-4217-af33-aae07198ca07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 16 12:55:04.952312 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:04.952261    2577 patch_prober.go:28] interesting pod/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 16 12:55:04.952839 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:04.952340    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz" podUID="99b70637-81ce-4503-bc3d-de5dc69e7b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 16 12:55:05.027043 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:05.026973    2577 patch_prober.go:28] interesting pod/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Sep 16 12:55:05.027225 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:05.027059    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs" podUID="462e0727-223b-4217-af33-aae07198ca07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 16 12:55:05.952296 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:05.952254    2577 patch_prober.go:28] interesting pod/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 16 12:55:05.952477 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:05.952339    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz" podUID="99b70637-81ce-4503-bc3d-de5dc69e7b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 16 12:55:06.026371 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:06.026314    2577 patch_prober.go:28] interesting pod/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Sep 16 12:55:06.026545 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:06.026396    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs" podUID="462e0727-223b-4217-af33-aae07198ca07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 16 12:55:06.952040 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:06.951983    2577 patch_prober.go:28] interesting pod/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 16 12:55:06.952259 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:06.952087    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz" podUID="99b70637-81ce-4503-bc3d-de5dc69e7b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 16 12:55:07.029735 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:07.029696    2577 patch_prober.go:28] interesting pod/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Sep 16 12:55:07.030419 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:07.030382    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs" podUID="462e0727-223b-4217-af33-aae07198ca07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 16 12:55:07.954052 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:07.953519    2577 patch_prober.go:28] interesting pod/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 16 12:55:07.954052 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:07.953602    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz" podUID="99b70637-81ce-4503-bc3d-de5dc69e7b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 16 12:55:08.027792 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:08.027746    2577 patch_prober.go:28] interesting pod/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Sep 16 12:55:08.027994 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:08.027820    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs" podUID="462e0727-223b-4217-af33-aae07198ca07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 16 12:55:08.957612 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:08.957584    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"
Sep 16 12:55:08.958538 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:08.958513    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"
Sep 16 12:55:08.959542 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:08.959525    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"
Sep 16 12:55:09.043358 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:09.043324    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"
Sep 16 12:55:09.044921 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:09.044896    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"
Sep 16 12:55:09.058259 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:09.058238    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"
Sep 16 12:55:09.607245 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:09.607207    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-79dbf9e9-7d59c45f4-j5mv4"
Sep 16 12:55:10.152225 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:10.152188    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-89d6bf-85cfbc9694-fpzc9"]
Sep 16 12:55:10.156181 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:10.156149    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-89d6bf-85cfbc9694-fpzc9"]
Sep 16 12:55:10.156313 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:10.156240    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89d6bf-85cfbc9694-fpzc9"
Sep 16 12:55:10.159749 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:10.159621    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-89d6bf-dockercfg-xvwsq\""
Sep 16 12:55:10.159749 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:10.159703    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-89d6bf\""
Sep 16 12:55:10.258894 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:10.258664    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8cx8\" (UniqueName: \"kubernetes.io/projected/7bb299ab-9db6-44bf-820d-6711a15f9e2c-kube-api-access-b8cx8\") pod \"payload-pre-processing-e2e-inf-b-89d6bf-85cfbc9694-fpzc9\" (UID: \"7bb299ab-9db6-44bf-820d-6711a15f9e2c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89d6bf-85cfbc9694-fpzc9"
Sep 16 12:55:10.258894 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:10.258766    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bb299ab-9db6-44bf-820d-6711a15f9e2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-89d6bf-85cfbc9694-fpzc9\" (UID: \"7bb299ab-9db6-44bf-820d-6711a15f9e2c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89d6bf-85cfbc9694-fpzc9"
Sep 16 12:55:10.359584 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:10.359553    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8cx8\" (UniqueName: \"kubernetes.io/projected/7bb299ab-9db6-44bf-820d-6711a15f9e2c-kube-api-access-b8cx8\") pod \"payload-pre-processing-e2e-inf-b-89d6bf-85cfbc9694-fpzc9\" (UID: \"7bb299ab-9db6-44bf-820d-6711a15f9e2c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89d6bf-85cfbc9694-fpzc9"
Sep 16 12:55:10.359742 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:10.359616    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bb299ab-9db6-44bf-820d-6711a15f9e2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-89d6bf-85cfbc9694-fpzc9\" (UID: \"7bb299ab-9db6-44bf-820d-6711a15f9e2c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89d6bf-85cfbc9694-fpzc9"
Sep 16 12:55:10.360436 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:10.360412    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bb299ab-9db6-44bf-820d-6711a15f9e2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-89d6bf-85cfbc9694-fpzc9\" (UID: \"7bb299ab-9db6-44bf-820d-6711a15f9e2c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89d6bf-85cfbc9694-fpzc9"
Sep 16 12:55:10.368548 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:10.368526    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8cx8\" (UniqueName: \"kubernetes.io/projected/7bb299ab-9db6-44bf-820d-6711a15f9e2c-kube-api-access-b8cx8\") pod \"payload-pre-processing-e2e-inf-b-89d6bf-85cfbc9694-fpzc9\" (UID: \"7bb299ab-9db6-44bf-820d-6711a15f9e2c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89d6bf-85cfbc9694-fpzc9"
Sep 16 12:55:10.468347 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:10.468247    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89d6bf-85cfbc9694-fpzc9"
Sep 16 12:55:10.775211 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:10.775189    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-89d6bf-85cfbc9694-fpzc9"]
Sep 16 12:55:10.777582 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:55:10.777557    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb299ab_9db6_44bf_820d_6711a15f9e2c.slice/crio-df265a7f327a87fc6dc546dcb4ceac6b3d2dc0e313e323c25633c16340889db7 WatchSource:0}: Error finding container df265a7f327a87fc6dc546dcb4ceac6b3d2dc0e313e323c25633c16340889db7: Status 404 returned error can't find the container with id df265a7f327a87fc6dc546dcb4ceac6b3d2dc0e313e323c25633c16340889db7
Sep 16 12:55:11.378077 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:11.378039    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-79dbf9e9-kcq26"]
Sep 16 12:55:11.382553 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:11.382530    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-79dbf9e9-kcq26"
Sep 16 12:55:11.386302 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:11.386099    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-64zph\""
Sep 16 12:55:11.386560 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:11.386535    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:55:11.388279 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:11.388246    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-79dbf9e9-kcq26"]
Sep 16 12:55:11.473237 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:11.473202    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9b59025-c033-4ceb-be93-fed3d9de03b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-79dbf9e9-kcq26\" (UID: \"f9b59025-c033-4ceb-be93-fed3d9de03b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-79dbf9e9-kcq26"
Sep 16 12:55:11.574548 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:11.574511    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9b59025-c033-4ceb-be93-fed3d9de03b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-79dbf9e9-kcq26\" (UID: \"f9b59025-c033-4ceb-be93-fed3d9de03b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-79dbf9e9-kcq26"
Sep 16 12:55:11.575221 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:11.575199    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9b59025-c033-4ceb-be93-fed3d9de03b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-79dbf9e9-kcq26\" (UID: \"f9b59025-c033-4ceb-be93-fed3d9de03b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-79dbf9e9-kcq26"
Sep 16 12:55:11.707117 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:11.707013    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-79dbf9e9-kcq26"
Sep 16 12:55:11.736249 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:11.735018    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89d6bf-85cfbc9694-fpzc9" event={"ID":"7bb299ab-9db6-44bf-820d-6711a15f9e2c","Type":"ContainerStarted","Data":"3921a3eca355ca8d409b8edaaaebbb3e44000c552460af988c635ef70f3f257d"}
Sep 16 12:55:11.736249 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:11.736083    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89d6bf-85cfbc9694-fpzc9"
Sep 16 12:55:11.736249 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:11.736136    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89d6bf-85cfbc9694-fpzc9" event={"ID":"7bb299ab-9db6-44bf-820d-6711a15f9e2c","Type":"ContainerStarted","Data":"df265a7f327a87fc6dc546dcb4ceac6b3d2dc0e313e323c25633c16340889db7"}
Sep 16 12:55:11.758511 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:11.758389    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89d6bf-85cfbc9694-fpzc9" podStartSLOduration=1.758366179 podStartE2EDuration="1.758366179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:55:11.755988883 +0000 UTC m=+1018.203221526" watchObservedRunningTime="2026-09-16 12:55:11.758366179 +0000 UTC m=+1018.205598823"
Sep 16 12:55:12.004689 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:12.004655    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-79dbf9e9-kcq26"]
Sep 16 12:55:12.007891 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:55:12.007844    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b59025_c033_4ceb_be93_fed3d9de03b0.slice/crio-f2b2f33a79a548dccd4153dad306133000467d50cf4c9b0a16f4d2d4aa05f257 WatchSource:0}: Error finding container f2b2f33a79a548dccd4153dad306133000467d50cf4c9b0a16f4d2d4aa05f257: Status 404 returned error can't find the container with id f2b2f33a79a548dccd4153dad306133000467d50cf4c9b0a16f4d2d4aa05f257
Sep 16 12:55:12.743354 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:12.743313    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-79dbf9e9-kcq26" event={"ID":"f9b59025-c033-4ceb-be93-fed3d9de03b0","Type":"ContainerStarted","Data":"f2b2f33a79a548dccd4153dad306133000467d50cf4c9b0a16f4d2d4aa05f257"}
Sep 16 12:55:14.759295 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:14.759257    2577 generic.go:356] "Generic (PLEG): container finished" podID="b9a5c65e-485d-480a-9cb4-2e306d451074" containerID="37f90da39d54be6cf90bc5e35155aeaf16f97f8534096e50d2407265f40af0b1" exitCode=0
Sep 16 12:55:14.759735 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:14.759336    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j" event={"ID":"b9a5c65e-485d-480a-9cb4-2e306d451074","Type":"ContainerDied","Data":"37f90da39d54be6cf90bc5e35155aeaf16f97f8534096e50d2407265f40af0b1"}
Sep 16 12:55:14.759829 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:14.759811    2577 scope.go:117] "RemoveContainer" containerID="37f90da39d54be6cf90bc5e35155aeaf16f97f8534096e50d2407265f40af0b1"
Sep 16 12:55:14.761085 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:14.761064    2577 generic.go:356] "Generic (PLEG): container finished" podID="1adf6c30-6569-40a5-ba22-caf26b38f80a" containerID="d960f8e232219bba39dd1c3f1886c89cff0d829e78db30c74539978576ab7aaf" exitCode=0
Sep 16 12:55:14.761202 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:14.761087    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm" event={"ID":"1adf6c30-6569-40a5-ba22-caf26b38f80a","Type":"ContainerDied","Data":"d960f8e232219bba39dd1c3f1886c89cff0d829e78db30c74539978576ab7aaf"}
Sep 16 12:55:14.761202 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:14.761065    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:55:14.761288 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:14.761245    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:55:14.761331 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:14.761307    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:55:14.761438 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:14.761425    2577 scope.go:117] "RemoveContainer" containerID="d960f8e232219bba39dd1c3f1886c89cff0d829e78db30c74539978576ab7aaf"
Sep 16 12:55:14.762461 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:14.762435    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:55:14.762561 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:14.762494    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:55:14.762561 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:14.762523    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:55:15.769656 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:15.769622    2577 generic.go:356] "Generic (PLEG): container finished" podID="f9b59025-c033-4ceb-be93-fed3d9de03b0" containerID="f5c41edf4b69f07849bf3e1f4916e2ada130140592ad219e635529a4ed49a25e" exitCode=0
Sep 16 12:55:15.770120 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:15.769714    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-79dbf9e9-kcq26" event={"ID":"f9b59025-c033-4ceb-be93-fed3d9de03b0","Type":"ContainerDied","Data":"f5c41edf4b69f07849bf3e1f4916e2ada130140592ad219e635529a4ed49a25e"}
Sep 16 12:55:15.771751 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:15.771726    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j" event={"ID":"b9a5c65e-485d-480a-9cb4-2e306d451074","Type":"ContainerStarted","Data":"6fe5277ed97839943af973ffab63db2adb1e1317cc3ea84e4f95653c76c2f6f4"}
Sep 16 12:55:15.773594 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:15.773570    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm" event={"ID":"1adf6c30-6569-40a5-ba22-caf26b38f80a","Type":"ContainerStarted","Data":"43f1c73b7bb3b639889ff63773e78b18431d1d194644763856a706aa5922b4c4"}
Sep 16 12:55:15.830781 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:15.830747    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"
Sep 16 12:55:15.830994 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:15.830974    2577 patch_prober.go:28] interesting pod/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Sep 16 12:55:15.831075 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:15.831038    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j" podUID="b9a5c65e-485d-480a-9cb4-2e306d451074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Sep 16 12:55:16.487943 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:16.487902    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"
Sep 16 12:55:16.488186 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:16.488075    2577 patch_prober.go:28] interesting pod/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 16 12:55:16.488186 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:16.488139    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm" podUID="1adf6c30-6569-40a5-ba22-caf26b38f80a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 16 12:55:16.833054 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:16.832394    2577 patch_prober.go:28] interesting pod/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Sep 16 12:55:16.833054 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:16.832454    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j" podUID="b9a5c65e-485d-480a-9cb4-2e306d451074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Sep 16 12:55:17.053667 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:17.053593    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-79dbf9e9-kcq26"
Sep 16 12:55:17.250461 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:17.250432    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9b59025-c033-4ceb-be93-fed3d9de03b0-maas-api-service-ca\") pod \"f9b59025-c033-4ceb-be93-fed3d9de03b0\" (UID: \"f9b59025-c033-4ceb-be93-fed3d9de03b0\") "
Sep 16 12:55:17.250816 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:17.250784    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9b59025-c033-4ceb-be93-fed3d9de03b0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f9b59025-c033-4ceb-be93-fed3d9de03b0" (UID: "f9b59025-c033-4ceb-be93-fed3d9de03b0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:55:17.251237 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:17.251216    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9b59025-c033-4ceb-be93-fed3d9de03b0-maas-api-service-ca\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:55:17.498125 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:17.492337    2577 patch_prober.go:28] interesting pod/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 16 12:55:17.498125 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:17.492395    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm" podUID="1adf6c30-6569-40a5-ba22-caf26b38f80a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 16 12:55:17.787734 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:17.787691    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-79dbf9e9-kcq26" event={"ID":"f9b59025-c033-4ceb-be93-fed3d9de03b0","Type":"ContainerDied","Data":"f2b2f33a79a548dccd4153dad306133000467d50cf4c9b0a16f4d2d4aa05f257"}
Sep 16 12:55:17.787949 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:17.787766    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2b2f33a79a548dccd4153dad306133000467d50cf4c9b0a16f4d2d4aa05f257"
Sep 16 12:55:17.787949 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:17.787832    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-79dbf9e9-kcq26"
Sep 16 12:55:17.831947 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:17.831494    2577 patch_prober.go:28] interesting pod/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Sep 16 12:55:17.831947 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:17.831559    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j" podUID="b9a5c65e-485d-480a-9cb4-2e306d451074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Sep 16 12:55:18.487787 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:18.487751    2577 patch_prober.go:28] interesting pod/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 16 12:55:18.488394 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:18.488359    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm" podUID="1adf6c30-6569-40a5-ba22-caf26b38f80a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 16 12:55:18.831257 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:18.831214    2577 patch_prober.go:28] interesting pod/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Sep 16 12:55:18.831567 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:18.831534    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j" podUID="b9a5c65e-485d-480a-9cb4-2e306d451074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Sep 16 12:55:19.488469 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:19.488423    2577 patch_prober.go:28] interesting pod/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 16 12:55:19.488955 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:19.488495    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm" podUID="1adf6c30-6569-40a5-ba22-caf26b38f80a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 16 12:55:19.831969 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:19.831501    2577 patch_prober.go:28] interesting pod/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Sep 16 12:55:19.831969 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:19.831569    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j" podUID="b9a5c65e-485d-480a-9cb4-2e306d451074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Sep 16 12:55:20.487599 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:20.487561    2577 patch_prober.go:28] interesting pod/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 16 12:55:20.487892 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:20.487859    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm" podUID="1adf6c30-6569-40a5-ba22-caf26b38f80a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 16 12:55:20.831059 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:20.830997    2577 patch_prober.go:28] interesting pod/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Sep 16 12:55:20.831619 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:20.831587    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j" podUID="b9a5c65e-485d-480a-9cb4-2e306d451074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Sep 16 12:55:21.488894 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:21.488843    2577 patch_prober.go:28] interesting pod/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 16 12:55:21.489211 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:21.489178    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm" podUID="1adf6c30-6569-40a5-ba22-caf26b38f80a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 16 12:55:21.547931 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:21.547876    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"
Sep 16 12:55:21.548229 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:21.548211    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"
Sep 16 12:55:21.835933 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:21.835907    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"
Sep 16 12:55:21.837345 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:21.837323    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"
Sep 16 12:55:22.033420 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:22.033380    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-79dbf9e9-7d59c45f4-j5mv4"]
Sep 16 12:55:22.034576 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:22.034520    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-79dbf9e9-7d59c45f4-j5mv4" podUID="9a7d8ce4-7b7f-4df0-b89f-24f52d065495" containerName="payload-pre-processing" containerID="cri-o://4118ebc57649ba1d249e612f29253ffc56ee0c4e69e1d7d0c10c285aa597dc82" gracePeriod=30
Sep 16 12:55:22.172172 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:22.172138    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-79dbf9e9-kcq26"]
Sep 16 12:55:22.180743 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:22.180677    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-79dbf9e9-kcq26"]
Sep 16 12:55:22.329071 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:22.327120    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"]
Sep 16 12:55:22.372506 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:22.372483    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-79dbf9e9-7d59c45f4-j5mv4"
Sep 16 12:55:22.440554 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:22.440446    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7d8ce4-7b7f-4df0-b89f-24f52d065495-plugins-config-volume\") pod \"9a7d8ce4-7b7f-4df0-b89f-24f52d065495\" (UID: \"9a7d8ce4-7b7f-4df0-b89f-24f52d065495\") "
Sep 16 12:55:22.440774 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:22.440559    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d44qq\" (UniqueName: \"kubernetes.io/projected/9a7d8ce4-7b7f-4df0-b89f-24f52d065495-kube-api-access-d44qq\") pod \"9a7d8ce4-7b7f-4df0-b89f-24f52d065495\" (UID: \"9a7d8ce4-7b7f-4df0-b89f-24f52d065495\") "
Sep 16 12:55:22.441664 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:22.441627    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a7d8ce4-7b7f-4df0-b89f-24f52d065495-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9a7d8ce4-7b7f-4df0-b89f-24f52d065495" (UID: "9a7d8ce4-7b7f-4df0-b89f-24f52d065495"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:55:22.444837 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:22.444806    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a7d8ce4-7b7f-4df0-b89f-24f52d065495-kube-api-access-d44qq" (OuterVolumeSpecName: "kube-api-access-d44qq") pod "9a7d8ce4-7b7f-4df0-b89f-24f52d065495" (UID: "9a7d8ce4-7b7f-4df0-b89f-24f52d065495"). InnerVolumeSpecName "kube-api-access-d44qq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:55:22.504759 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:22.504717    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"
Sep 16 12:55:22.507290 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:22.507229    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"
Sep 16 12:55:22.541833 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:22.541798    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7d8ce4-7b7f-4df0-b89f-24f52d065495-plugins-config-volume\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:55:22.541833 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:22.541831    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d44qq\" (UniqueName: \"kubernetes.io/projected/9a7d8ce4-7b7f-4df0-b89f-24f52d065495-kube-api-access-d44qq\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:55:22.745986 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:22.745900    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89d6bf-85cfbc9694-fpzc9"
Sep 16 12:55:22.837961 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:22.837932    2577 generic.go:356] "Generic (PLEG): container finished" podID="9a7d8ce4-7b7f-4df0-b89f-24f52d065495" containerID="4118ebc57649ba1d249e612f29253ffc56ee0c4e69e1d7d0c10c285aa597dc82" exitCode=0
Sep 16 12:55:22.838898 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:22.838017    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-79dbf9e9-7d59c45f4-j5mv4"
Sep 16 12:55:22.838898 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:22.838014    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-79dbf9e9-7d59c45f4-j5mv4" event={"ID":"9a7d8ce4-7b7f-4df0-b89f-24f52d065495","Type":"ContainerDied","Data":"4118ebc57649ba1d249e612f29253ffc56ee0c4e69e1d7d0c10c285aa597dc82"}
Sep 16 12:55:22.838898 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:22.838671    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-79dbf9e9-7d59c45f4-j5mv4" event={"ID":"9a7d8ce4-7b7f-4df0-b89f-24f52d065495","Type":"ContainerDied","Data":"1c73ccec8e1567f4d039bb2293d5f7b61542612d2ce91e0073d3e3d4c8348b58"}
Sep 16 12:55:22.838898 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:22.838696    2577 scope.go:117] "RemoveContainer" containerID="4118ebc57649ba1d249e612f29253ffc56ee0c4e69e1d7d0c10c285aa597dc82"
Sep 16 12:55:22.840451 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:22.840337    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j" secret="" err="secret \"e2e-ait-79dbf9e9-openshift-default-dockercfg-hpls9\" not found"
Sep 16 12:55:22.850787 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:22.850773    2577 scope.go:117] "RemoveContainer" containerID="4118ebc57649ba1d249e612f29253ffc56ee0c4e69e1d7d0c10c285aa597dc82"
Sep 16 12:55:22.851124 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:55:22.851099    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4118ebc57649ba1d249e612f29253ffc56ee0c4e69e1d7d0c10c285aa597dc82\": container with ID starting with 4118ebc57649ba1d249e612f29253ffc56ee0c4e69e1d7d0c10c285aa597dc82 not found: ID does not exist" containerID="4118ebc57649ba1d249e612f29253ffc56ee0c4e69e1d7d0c10c285aa597dc82"
Sep 16 12:55:22.851173 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:22.851138    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4118ebc57649ba1d249e612f29253ffc56ee0c4e69e1d7d0c10c285aa597dc82"} err="failed to get container status \"4118ebc57649ba1d249e612f29253ffc56ee0c4e69e1d7d0c10c285aa597dc82\": rpc error: code = NotFound desc = could not find container \"4118ebc57649ba1d249e612f29253ffc56ee0c4e69e1d7d0c10c285aa597dc82\": container with ID starting with 4118ebc57649ba1d249e612f29253ffc56ee0c4e69e1d7d0c10c285aa597dc82 not found: ID does not exist"
Sep 16 12:55:22.886732 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:22.886702    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-79dbf9e9-7d59c45f4-j5mv4"]
Sep 16 12:55:22.908007 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:22.907973    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-79dbf9e9-7d59c45f4-j5mv4"]
Sep 16 12:55:23.312660 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.312623    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"]
Sep 16 12:55:23.313641 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.313617    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a7d8ce4-7b7f-4df0-b89f-24f52d065495" containerName="payload-pre-processing"
Sep 16 12:55:23.313794 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.313782    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a7d8ce4-7b7f-4df0-b89f-24f52d065495" containerName="payload-pre-processing"
Sep 16 12:55:23.313943 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.313931    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9b59025-c033-4ceb-be93-fed3d9de03b0" containerName="revoke-keys"
Sep 16 12:55:23.314024 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.314014    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9b59025-c033-4ceb-be93-fed3d9de03b0" containerName="revoke-keys"
Sep 16 12:55:23.314289 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.314275    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a7d8ce4-7b7f-4df0-b89f-24f52d065495" containerName="payload-pre-processing"
Sep 16 12:55:23.314383 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.314373    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9b59025-c033-4ceb-be93-fed3d9de03b0" containerName="revoke-keys"
Sep 16 12:55:23.319917 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.319898    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"
Sep 16 12:55:23.325457 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.325436    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-e3535999-gw-openshift-default-dockercfg-jsznk\""
Sep 16 12:55:23.328104 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.326493    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"]
Sep 16 12:55:23.351101 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.351076    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ad95431-ff85-4e77-9e49-c222239e9bce-workload-certs\") pod \"e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp\" (UID: \"7ad95431-ff85-4e77-9e49-c222239e9bce\") " pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"
Sep 16 12:55:23.351234 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.351105    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ad95431-ff85-4e77-9e49-c222239e9bce-istiod-ca-cert\") pod \"e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp\" (UID: \"7ad95431-ff85-4e77-9e49-c222239e9bce\") " pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"
Sep 16 12:55:23.351234 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.351141    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ad95431-ff85-4e77-9e49-c222239e9bce-istio-token\") pod \"e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp\" (UID: \"7ad95431-ff85-4e77-9e49-c222239e9bce\") " pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"
Sep 16 12:55:23.351234 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.351165    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvx6l\" (UniqueName: \"kubernetes.io/projected/7ad95431-ff85-4e77-9e49-c222239e9bce-kube-api-access-gvx6l\") pod \"e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp\" (UID: \"7ad95431-ff85-4e77-9e49-c222239e9bce\") " pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"
Sep 16 12:55:23.351234 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.351217    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad95431-ff85-4e77-9e49-c222239e9bce-credential-socket\") pod \"e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp\" (UID: \"7ad95431-ff85-4e77-9e49-c222239e9bce\") " pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"
Sep 16 12:55:23.351464 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.351236    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ad95431-ff85-4e77-9e49-c222239e9bce-istio-data\") pod \"e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp\" (UID: \"7ad95431-ff85-4e77-9e49-c222239e9bce\") " pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"
Sep 16 12:55:23.351464 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.351260    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ad95431-ff85-4e77-9e49-c222239e9bce-istio-envoy\") pod \"e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp\" (UID: \"7ad95431-ff85-4e77-9e49-c222239e9bce\") " pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"
Sep 16 12:55:23.351464 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.351399    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad95431-ff85-4e77-9e49-c222239e9bce-workload-socket\") pod \"e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp\" (UID: \"7ad95431-ff85-4e77-9e49-c222239e9bce\") " pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"
Sep 16 12:55:23.351623 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.351495    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ad95431-ff85-4e77-9e49-c222239e9bce-istio-podinfo\") pod \"e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp\" (UID: \"7ad95431-ff85-4e77-9e49-c222239e9bce\") " pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"
Sep 16 12:55:23.452570 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.452529    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad95431-ff85-4e77-9e49-c222239e9bce-credential-socket\") pod \"e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp\" (UID: \"7ad95431-ff85-4e77-9e49-c222239e9bce\") " pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"
Sep 16 12:55:23.453271 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.453242    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad95431-ff85-4e77-9e49-c222239e9bce-credential-socket\") pod \"e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp\" (UID: \"7ad95431-ff85-4e77-9e49-c222239e9bce\") " pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"
Sep 16 12:55:23.453455 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.453295    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ad95431-ff85-4e77-9e49-c222239e9bce-istio-data\") pod \"e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp\" (UID: \"7ad95431-ff85-4e77-9e49-c222239e9bce\") " pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"
Sep 16 12:55:23.453627 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.453606    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ad95431-ff85-4e77-9e49-c222239e9bce-istio-envoy\") pod \"e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp\" (UID: \"7ad95431-ff85-4e77-9e49-c222239e9bce\") " pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"
Sep 16 12:55:23.453825 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.453809    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad95431-ff85-4e77-9e49-c222239e9bce-workload-socket\") pod \"e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp\" (UID: \"7ad95431-ff85-4e77-9e49-c222239e9bce\") " pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"
Sep 16 12:55:23.453958 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.453943    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ad95431-ff85-4e77-9e49-c222239e9bce-istio-podinfo\") pod \"e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp\" (UID: \"7ad95431-ff85-4e77-9e49-c222239e9bce\") " pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"
Sep 16 12:55:23.454131 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.454114    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ad95431-ff85-4e77-9e49-c222239e9bce-workload-certs\") pod \"e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp\" (UID: \"7ad95431-ff85-4e77-9e49-c222239e9bce\") " pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"
Sep 16 12:55:23.454253 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.454240    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ad95431-ff85-4e77-9e49-c222239e9bce-istiod-ca-cert\") pod \"e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp\" (UID: \"7ad95431-ff85-4e77-9e49-c222239e9bce\") " pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"
Sep 16 12:55:23.454348 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.454336    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ad95431-ff85-4e77-9e49-c222239e9bce-istio-token\") pod \"e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp\" (UID: \"7ad95431-ff85-4e77-9e49-c222239e9bce\") " pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"
Sep 16 12:55:23.454450 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.454438    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvx6l\" (UniqueName: \"kubernetes.io/projected/7ad95431-ff85-4e77-9e49-c222239e9bce-kube-api-access-gvx6l\") pod \"e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp\" (UID: \"7ad95431-ff85-4e77-9e49-c222239e9bce\") " pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"
Sep 16 12:55:23.454991 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.453630    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ad95431-ff85-4e77-9e49-c222239e9bce-istio-data\") pod \"e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp\" (UID: \"7ad95431-ff85-4e77-9e49-c222239e9bce\") " pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"
Sep 16 12:55:23.455431 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.455399    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ad95431-ff85-4e77-9e49-c222239e9bce-istiod-ca-cert\") pod \"e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp\" (UID: \"7ad95431-ff85-4e77-9e49-c222239e9bce\") " pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"
Sep 16 12:55:23.455598 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.454186    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad95431-ff85-4e77-9e49-c222239e9bce-workload-socket\") pod \"e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp\" (UID: \"7ad95431-ff85-4e77-9e49-c222239e9bce\") " pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"
Sep 16 12:55:23.459185 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.455730    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ad95431-ff85-4e77-9e49-c222239e9bce-workload-certs\") pod \"e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp\" (UID: \"7ad95431-ff85-4e77-9e49-c222239e9bce\") " pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"
Sep 16 12:55:23.459689 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.459662    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ad95431-ff85-4e77-9e49-c222239e9bce-istio-podinfo\") pod \"e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp\" (UID: \"7ad95431-ff85-4e77-9e49-c222239e9bce\") " pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"
Sep 16 12:55:23.459960 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.459893    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ad95431-ff85-4e77-9e49-c222239e9bce-istio-envoy\") pod \"e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp\" (UID: \"7ad95431-ff85-4e77-9e49-c222239e9bce\") " pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"
Sep 16 12:55:23.463465 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.463441    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ad95431-ff85-4e77-9e49-c222239e9bce-istio-token\") pod \"e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp\" (UID: \"7ad95431-ff85-4e77-9e49-c222239e9bce\") " pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"
Sep 16 12:55:23.463548 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.463525    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvx6l\" (UniqueName: \"kubernetes.io/projected/7ad95431-ff85-4e77-9e49-c222239e9bce-kube-api-access-gvx6l\") pod \"e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp\" (UID: \"7ad95431-ff85-4e77-9e49-c222239e9bce\") " pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"
Sep 16 12:55:23.637903 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.637810    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"
Sep 16 12:55:23.845369 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.845330    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j" podUID="b9a5c65e-485d-480a-9cb4-2e306d451074" containerName="istio-proxy" containerID="cri-o://6fe5277ed97839943af973ffab63db2adb1e1317cc3ea84e4f95653c76c2f6f4" gracePeriod=30
Sep 16 12:55:23.959484 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.959455    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"]
Sep 16 12:55:23.961117 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:55:23.961087    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad95431_ff85_4e77_9e49_c222239e9bce.slice/crio-c87382ecf5a047e267ec0df29516fdbe4ed48b0bfa0ac6c1e72d8a6ab6c544b9 WatchSource:0}: Error finding container c87382ecf5a047e267ec0df29516fdbe4ed48b0bfa0ac6c1e72d8a6ab6c544b9: Status 404 returned error can't find the container with id c87382ecf5a047e267ec0df29516fdbe4ed48b0bfa0ac6c1e72d8a6ab6c544b9
Sep 16 12:55:23.963804 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.963772    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:55:23.963897 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.963854    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:55:23.963943 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:23.963929    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:55:24.056611 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:24.056582    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a7d8ce4-7b7f-4df0-b89f-24f52d065495" path="/var/lib/kubelet/pods/9a7d8ce4-7b7f-4df0-b89f-24f52d065495/volumes"
Sep 16 12:55:24.057078 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:24.057058    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9b59025-c033-4ceb-be93-fed3d9de03b0" path="/var/lib/kubelet/pods/f9b59025-c033-4ceb-be93-fed3d9de03b0/volumes"
Sep 16 12:55:24.852924 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:24.852882    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp" event={"ID":"7ad95431-ff85-4e77-9e49-c222239e9bce","Type":"ContainerStarted","Data":"901d59daecf83099c60bd91e23cd8763f6a7aaf29ec2ec89056d5a76ee80dec7"}
Sep 16 12:55:24.853348 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:24.852980    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp" event={"ID":"7ad95431-ff85-4e77-9e49-c222239e9bce","Type":"ContainerStarted","Data":"c87382ecf5a047e267ec0df29516fdbe4ed48b0bfa0ac6c1e72d8a6ab6c544b9"}
Sep 16 12:55:24.875399 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:24.875355    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp" podStartSLOduration=1.875340056 podStartE2EDuration="1.875340056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:55:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:55:24.874230889 +0000 UTC m=+1031.321463556" watchObservedRunningTime="2026-09-16 12:55:24.875340056 +0000 UTC m=+1031.322572694"
Sep 16 12:55:25.638151 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:25.638112    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"
Sep 16 12:55:25.643770 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:25.643742    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"
Sep 16 12:55:25.859116 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:25.859078    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"
Sep 16 12:55:25.859971 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:25.859949    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"
Sep 16 12:55:27.874771 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:27.874664    2577 generic.go:356] "Generic (PLEG): container finished" podID="10ebb745-2b51-4ac4-8834-66601208abdb" containerID="15d28b872da26825994a5e63c24434579b6629a7aa70ab960c87994de2843d68" exitCode=0
Sep 16 12:55:27.875290 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:27.874995    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888" event={"ID":"10ebb745-2b51-4ac4-8834-66601208abdb","Type":"ContainerDied","Data":"15d28b872da26825994a5e63c24434579b6629a7aa70ab960c87994de2843d68"}
Sep 16 12:55:27.875553 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:27.875527    2577 scope.go:117] "RemoveContainer" containerID="15d28b872da26825994a5e63c24434579b6629a7aa70ab960c87994de2843d68"
Sep 16 12:55:27.877057 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:27.876994    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:55:27.877168 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:27.877102    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:55:27.877168 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:27.877149    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:55:28.882800 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:28.882764    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888" event={"ID":"10ebb745-2b51-4ac4-8834-66601208abdb","Type":"ContainerStarted","Data":"44155e108f624e9db292ca763d86d9643dbb1ead640e06727dbd72aadd49522e"}
Sep 16 12:55:28.986753 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:28.986720    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"
Sep 16 12:55:28.993053 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:28.991174    2577 patch_prober.go:28] interesting pod/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 16 12:55:28.993053 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:28.991262    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888" podUID="10ebb745-2b51-4ac4-8834-66601208abdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 16 12:55:29.161149 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.161118    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"
Sep 16 12:55:29.227529 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.226595    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxwck\" (UniqueName: \"kubernetes.io/projected/b9a5c65e-485d-480a-9cb4-2e306d451074-kube-api-access-pxwck\") pod \"b9a5c65e-485d-480a-9cb4-2e306d451074\" (UID: \"b9a5c65e-485d-480a-9cb4-2e306d451074\") "
Sep 16 12:55:29.227529 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.226638    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9a5c65e-485d-480a-9cb4-2e306d451074-istio-token\") pod \"b9a5c65e-485d-480a-9cb4-2e306d451074\" (UID: \"b9a5c65e-485d-480a-9cb4-2e306d451074\") "
Sep 16 12:55:29.227529 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.226693    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9a5c65e-485d-480a-9cb4-2e306d451074-istiod-ca-cert\") pod \"b9a5c65e-485d-480a-9cb4-2e306d451074\" (UID: \"b9a5c65e-485d-480a-9cb4-2e306d451074\") "
Sep 16 12:55:29.227529 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.226714    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9a5c65e-485d-480a-9cb4-2e306d451074-istio-data\") pod \"b9a5c65e-485d-480a-9cb4-2e306d451074\" (UID: \"b9a5c65e-485d-480a-9cb4-2e306d451074\") "
Sep 16 12:55:29.227529 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.226731    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9a5c65e-485d-480a-9cb4-2e306d451074-istio-envoy\") pod \"b9a5c65e-485d-480a-9cb4-2e306d451074\" (UID: \"b9a5c65e-485d-480a-9cb4-2e306d451074\") "
Sep 16 12:55:29.227529 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.226755    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a5c65e-485d-480a-9cb4-2e306d451074-credential-socket\") pod \"b9a5c65e-485d-480a-9cb4-2e306d451074\" (UID: \"b9a5c65e-485d-480a-9cb4-2e306d451074\") "
Sep 16 12:55:29.227529 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.226787    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9a5c65e-485d-480a-9cb4-2e306d451074-istio-podinfo\") pod \"b9a5c65e-485d-480a-9cb4-2e306d451074\" (UID: \"b9a5c65e-485d-480a-9cb4-2e306d451074\") "
Sep 16 12:55:29.227529 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.226812    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a5c65e-485d-480a-9cb4-2e306d451074-workload-socket\") pod \"b9a5c65e-485d-480a-9cb4-2e306d451074\" (UID: \"b9a5c65e-485d-480a-9cb4-2e306d451074\") "
Sep 16 12:55:29.227529 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.226837    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9a5c65e-485d-480a-9cb4-2e306d451074-workload-certs\") pod \"b9a5c65e-485d-480a-9cb4-2e306d451074\" (UID: \"b9a5c65e-485d-480a-9cb4-2e306d451074\") "
Sep 16 12:55:29.227529 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.226996    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9a5c65e-485d-480a-9cb4-2e306d451074-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b9a5c65e-485d-480a-9cb4-2e306d451074" (UID: "b9a5c65e-485d-480a-9cb4-2e306d451074"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:55:29.227529 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.227122    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a5c65e-485d-480a-9cb4-2e306d451074-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b9a5c65e-485d-480a-9cb4-2e306d451074" (UID: "b9a5c65e-485d-480a-9cb4-2e306d451074"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:55:29.227529 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.227156    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9a5c65e-485d-480a-9cb4-2e306d451074-istiod-ca-cert\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:55:29.227529 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.227204    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a5c65e-485d-480a-9cb4-2e306d451074-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b9a5c65e-485d-480a-9cb4-2e306d451074" (UID: "b9a5c65e-485d-480a-9cb4-2e306d451074"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:55:29.227529 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.227365    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a5c65e-485d-480a-9cb4-2e306d451074-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b9a5c65e-485d-480a-9cb4-2e306d451074" (UID: "b9a5c65e-485d-480a-9cb4-2e306d451074"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:55:29.227529 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.227310    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a5c65e-485d-480a-9cb4-2e306d451074-istio-data" (OuterVolumeSpecName: "istio-data") pod "b9a5c65e-485d-480a-9cb4-2e306d451074" (UID: "b9a5c65e-485d-480a-9cb4-2e306d451074"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:55:29.229269 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.229234    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a5c65e-485d-480a-9cb4-2e306d451074-istio-token" (OuterVolumeSpecName: "istio-token") pod "b9a5c65e-485d-480a-9cb4-2e306d451074" (UID: "b9a5c65e-485d-480a-9cb4-2e306d451074"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:55:29.229563 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.229540    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a5c65e-485d-480a-9cb4-2e306d451074-kube-api-access-pxwck" (OuterVolumeSpecName: "kube-api-access-pxwck") pod "b9a5c65e-485d-480a-9cb4-2e306d451074" (UID: "b9a5c65e-485d-480a-9cb4-2e306d451074"). InnerVolumeSpecName "kube-api-access-pxwck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:55:29.230014 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.229996    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a5c65e-485d-480a-9cb4-2e306d451074-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b9a5c65e-485d-480a-9cb4-2e306d451074" (UID: "b9a5c65e-485d-480a-9cb4-2e306d451074"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:55:29.230436 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.230415    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9a5c65e-485d-480a-9cb4-2e306d451074-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b9a5c65e-485d-480a-9cb4-2e306d451074" (UID: "b9a5c65e-485d-480a-9cb4-2e306d451074"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:55:29.327984 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.327948    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxwck\" (UniqueName: \"kubernetes.io/projected/b9a5c65e-485d-480a-9cb4-2e306d451074-kube-api-access-pxwck\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:55:29.327984 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.327978    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9a5c65e-485d-480a-9cb4-2e306d451074-istio-token\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:55:29.327984 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.327986    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9a5c65e-485d-480a-9cb4-2e306d451074-istio-data\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:55:29.327984 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.327994    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9a5c65e-485d-480a-9cb4-2e306d451074-istio-envoy\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:55:29.328359 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.328003    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a5c65e-485d-480a-9cb4-2e306d451074-credential-socket\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:55:29.328359 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.328011    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9a5c65e-485d-480a-9cb4-2e306d451074-istio-podinfo\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:55:29.328359 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.328018    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a5c65e-485d-480a-9cb4-2e306d451074-workload-socket\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:55:29.328359 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.328044    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9a5c65e-485d-480a-9cb4-2e306d451074-workload-certs\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:55:29.893986 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.893942    2577 generic.go:356] "Generic (PLEG): container finished" podID="b9a5c65e-485d-480a-9cb4-2e306d451074" containerID="6fe5277ed97839943af973ffab63db2adb1e1317cc3ea84e4f95653c76c2f6f4" exitCode=0
Sep 16 12:55:29.894960 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.894933    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"
Sep 16 12:55:29.901418 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.901191    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j" event={"ID":"b9a5c65e-485d-480a-9cb4-2e306d451074","Type":"ContainerDied","Data":"6fe5277ed97839943af973ffab63db2adb1e1317cc3ea84e4f95653c76c2f6f4"}
Sep 16 12:55:29.901418 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.901258    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j" event={"ID":"b9a5c65e-485d-480a-9cb4-2e306d451074","Type":"ContainerDied","Data":"05672875a1bd265b888103edbaad7fd5a4de446e59b19b8a99c6c8f9d15dec56"}
Sep 16 12:55:29.901418 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.901291    2577 scope.go:117] "RemoveContainer" containerID="6fe5277ed97839943af973ffab63db2adb1e1317cc3ea84e4f95653c76c2f6f4"
Sep 16 12:55:29.920296 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.920278    2577 scope.go:117] "RemoveContainer" containerID="37f90da39d54be6cf90bc5e35155aeaf16f97f8534096e50d2407265f40af0b1"
Sep 16 12:55:29.927477 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.927454    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"]
Sep 16 12:55:29.930777 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.930753    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-79dbf9e9-openshift-default-669df67698-6mk6j"]
Sep 16 12:55:29.931217 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.931198    2577 scope.go:117] "RemoveContainer" containerID="6fe5277ed97839943af973ffab63db2adb1e1317cc3ea84e4f95653c76c2f6f4"
Sep 16 12:55:29.931502 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:55:29.931482    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fe5277ed97839943af973ffab63db2adb1e1317cc3ea84e4f95653c76c2f6f4\": container with ID starting with 6fe5277ed97839943af973ffab63db2adb1e1317cc3ea84e4f95653c76c2f6f4 not found: ID does not exist" containerID="6fe5277ed97839943af973ffab63db2adb1e1317cc3ea84e4f95653c76c2f6f4"
Sep 16 12:55:29.931562 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.931512    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fe5277ed97839943af973ffab63db2adb1e1317cc3ea84e4f95653c76c2f6f4"} err="failed to get container status \"6fe5277ed97839943af973ffab63db2adb1e1317cc3ea84e4f95653c76c2f6f4\": rpc error: code = NotFound desc = could not find container \"6fe5277ed97839943af973ffab63db2adb1e1317cc3ea84e4f95653c76c2f6f4\": container with ID starting with 6fe5277ed97839943af973ffab63db2adb1e1317cc3ea84e4f95653c76c2f6f4 not found: ID does not exist"
Sep 16 12:55:29.931562 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.931529    2577 scope.go:117] "RemoveContainer" containerID="37f90da39d54be6cf90bc5e35155aeaf16f97f8534096e50d2407265f40af0b1"
Sep 16 12:55:29.931831 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:55:29.931814    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37f90da39d54be6cf90bc5e35155aeaf16f97f8534096e50d2407265f40af0b1\": container with ID starting with 37f90da39d54be6cf90bc5e35155aeaf16f97f8534096e50d2407265f40af0b1 not found: ID does not exist" containerID="37f90da39d54be6cf90bc5e35155aeaf16f97f8534096e50d2407265f40af0b1"
Sep 16 12:55:29.931885 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.931835    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37f90da39d54be6cf90bc5e35155aeaf16f97f8534096e50d2407265f40af0b1"} err="failed to get container status \"37f90da39d54be6cf90bc5e35155aeaf16f97f8534096e50d2407265f40af0b1\": rpc error: code = NotFound desc = could not find container \"37f90da39d54be6cf90bc5e35155aeaf16f97f8534096e50d2407265f40af0b1\": container with ID starting with 37f90da39d54be6cf90bc5e35155aeaf16f97f8534096e50d2407265f40af0b1 not found: ID does not exist"
Sep 16 12:55:29.986738 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.986708    2577 patch_prober.go:28] interesting pod/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 16 12:55:29.986827 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:29.986771    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888" podUID="10ebb745-2b51-4ac4-8834-66601208abdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 16 12:55:30.057686 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:30.057641    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9a5c65e-485d-480a-9cb4-2e306d451074" path="/var/lib/kubelet/pods/b9a5c65e-485d-480a-9cb4-2e306d451074/volumes"
Sep 16 12:55:30.917088 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:30.917051    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e3535999-bb8778658-f8p24"]
Sep 16 12:55:30.917726 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:30.917706    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9a5c65e-485d-480a-9cb4-2e306d451074" containerName="istio-proxy"
Sep 16 12:55:30.917801 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:30.917728    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9a5c65e-485d-480a-9cb4-2e306d451074" containerName="istio-proxy"
Sep 16 12:55:30.917801 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:30.917748    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9a5c65e-485d-480a-9cb4-2e306d451074" containerName="istio-proxy"
Sep 16 12:55:30.917801 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:30.917756    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9a5c65e-485d-480a-9cb4-2e306d451074" containerName="istio-proxy"
Sep 16 12:55:30.917958 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:30.917865    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9a5c65e-485d-480a-9cb4-2e306d451074" containerName="istio-proxy"
Sep 16 12:55:30.917958 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:30.917877    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9a5c65e-485d-480a-9cb4-2e306d451074" containerName="istio-proxy"
Sep 16 12:55:30.922756 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:30.922737    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e3535999-bb8778658-f8p24"
Sep 16 12:55:30.926306 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:30.926248    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-e3535999\""
Sep 16 12:55:30.927267 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:30.927133    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-e3535999-dockercfg-htfp4\""
Sep 16 12:55:30.928857 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:30.928806    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e3535999-bb8778658-f8p24"]
Sep 16 12:55:30.944545 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:30.944523    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcl8r\" (UniqueName: \"kubernetes.io/projected/9606b33d-d777-436c-87f0-4b9548132a60-kube-api-access-xcl8r\") pod \"payload-pre-processing-e2e-migrate-e3535999-bb8778658-f8p24\" (UID: \"9606b33d-d777-436c-87f0-4b9548132a60\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e3535999-bb8778658-f8p24"
Sep 16 12:55:30.944707 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:30.944688    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9606b33d-d777-436c-87f0-4b9548132a60-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e3535999-bb8778658-f8p24\" (UID: \"9606b33d-d777-436c-87f0-4b9548132a60\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e3535999-bb8778658-f8p24"
Sep 16 12:55:30.987328 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:30.987277    2577 patch_prober.go:28] interesting pod/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 16 12:55:30.987518 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:30.987348    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888" podUID="10ebb745-2b51-4ac4-8834-66601208abdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 16 12:55:31.045390 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:31.045355    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9606b33d-d777-436c-87f0-4b9548132a60-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e3535999-bb8778658-f8p24\" (UID: \"9606b33d-d777-436c-87f0-4b9548132a60\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e3535999-bb8778658-f8p24"
Sep 16 12:55:31.045390 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:31.045389    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcl8r\" (UniqueName: \"kubernetes.io/projected/9606b33d-d777-436c-87f0-4b9548132a60-kube-api-access-xcl8r\") pod \"payload-pre-processing-e2e-migrate-e3535999-bb8778658-f8p24\" (UID: \"9606b33d-d777-436c-87f0-4b9548132a60\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e3535999-bb8778658-f8p24"
Sep 16 12:55:31.046184 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:31.046159    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9606b33d-d777-436c-87f0-4b9548132a60-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e3535999-bb8778658-f8p24\" (UID: \"9606b33d-d777-436c-87f0-4b9548132a60\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e3535999-bb8778658-f8p24"
Sep 16 12:55:31.056234 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:31.056210    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcl8r\" (UniqueName: \"kubernetes.io/projected/9606b33d-d777-436c-87f0-4b9548132a60-kube-api-access-xcl8r\") pod \"payload-pre-processing-e2e-migrate-e3535999-bb8778658-f8p24\" (UID: \"9606b33d-d777-436c-87f0-4b9548132a60\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e3535999-bb8778658-f8p24"
Sep 16 12:55:31.234106 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:31.234010    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e3535999-bb8778658-f8p24"
Sep 16 12:55:31.524929 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:31.524904    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e3535999-bb8778658-f8p24"]
Sep 16 12:55:31.527278 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:55:31.527239    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9606b33d_d777_436c_87f0_4b9548132a60.slice/crio-c1fb2ddcfbe39cd0358cf8ed08e8bfb0deec1177f0ecb67ecc5c5afd0a10279e WatchSource:0}: Error finding container c1fb2ddcfbe39cd0358cf8ed08e8bfb0deec1177f0ecb67ecc5c5afd0a10279e: Status 404 returned error can't find the container with id c1fb2ddcfbe39cd0358cf8ed08e8bfb0deec1177f0ecb67ecc5c5afd0a10279e
Sep 16 12:55:31.920605 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:31.920497    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e3535999-bb8778658-f8p24" event={"ID":"9606b33d-d777-436c-87f0-4b9548132a60","Type":"ContainerStarted","Data":"0387eadd973a4608a78924aa10e2a29c820d92512a7006ece74bc559d4e6b8d2"}
Sep 16 12:55:31.920605 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:31.920562    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e3535999-bb8778658-f8p24" event={"ID":"9606b33d-d777-436c-87f0-4b9548132a60","Type":"ContainerStarted","Data":"c1fb2ddcfbe39cd0358cf8ed08e8bfb0deec1177f0ecb67ecc5c5afd0a10279e"}
Sep 16 12:55:31.920605 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:31.920604    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e3535999-bb8778658-f8p24"
Sep 16 12:55:31.941208 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:31.941147    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e3535999-bb8778658-f8p24" podStartSLOduration=1.941129259 podStartE2EDuration="1.941129259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:55:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:55:31.939325338 +0000 UTC m=+1038.386557980" watchObservedRunningTime="2026-09-16 12:55:31.941129259 +0000 UTC m=+1038.388361902"
Sep 16 12:55:31.987260 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:31.987222    2577 patch_prober.go:28] interesting pod/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 16 12:55:31.987442 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:31.987286    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888" podUID="10ebb745-2b51-4ac4-8834-66601208abdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 16 12:55:32.987446 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:32.987412    2577 patch_prober.go:28] interesting pod/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 16 12:55:32.988047 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:32.987995    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888" podUID="10ebb745-2b51-4ac4-8834-66601208abdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 16 12:55:33.646982 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:33.646933    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"
Sep 16 12:55:33.987776 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:33.987670    2577 patch_prober.go:28] interesting pod/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 16 12:55:33.987776 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:33.987739    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888" podUID="10ebb745-2b51-4ac4-8834-66601208abdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 16 12:55:34.992007 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:34.991976    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"
Sep 16 12:55:34.993095 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:34.993077    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"
Sep 16 12:55:42.928013 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:42.927984    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e3535999-bb8778658-f8p24"
Sep 16 12:55:48.014926 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:48.014891    2577 generic.go:356] "Generic (PLEG): container finished" podID="7ad95431-ff85-4e77-9e49-c222239e9bce" containerID="901d59daecf83099c60bd91e23cd8763f6a7aaf29ec2ec89056d5a76ee80dec7" exitCode=0
Sep 16 12:55:48.015354 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:48.014964    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp" event={"ID":"7ad95431-ff85-4e77-9e49-c222239e9bce","Type":"ContainerDied","Data":"901d59daecf83099c60bd91e23cd8763f6a7aaf29ec2ec89056d5a76ee80dec7"}
Sep 16 12:55:48.015447 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:48.015434    2577 scope.go:117] "RemoveContainer" containerID="901d59daecf83099c60bd91e23cd8763f6a7aaf29ec2ec89056d5a76ee80dec7"
Sep 16 12:55:48.016598 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:48.016575    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:55:48.016655 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:48.016640    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:55:48.016694 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:48.016683    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:55:49.023379 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:49.023343    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp" event={"ID":"7ad95431-ff85-4e77-9e49-c222239e9bce","Type":"ContainerStarted","Data":"21b42f9f58d03f45b00f2f36625f3479ca9fed10ec9f676060e4feaf541fc33e"}
Sep 16 12:55:49.466902 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:49.466810    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-490797/simulator-w0-490797-kserve-7fb7c79cf5-w9scf"]
Sep 16 12:55:49.470630 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:49.470607    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-490797/simulator-w0-490797-kserve-7fb7c79cf5-w9scf"
Sep 16 12:55:49.473593 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:49.473573    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-490797\"/\"kube-root-ca.crt\""
Sep 16 12:55:49.474803 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:49.474786    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-490797\"/\"openshift-service-ca.crt\""
Sep 16 12:55:49.474887 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:49.474786    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-490797\"/\"simulator-w0-490797-kserve-self-signed-certs\""
Sep 16 12:55:49.475082 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:49.475066    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-490797\"/\"default-dockercfg-vb4tt\""
Sep 16 12:55:49.481354 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:49.481333    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-490797/simulator-w0-490797-kserve-7fb7c79cf5-w9scf"]
Sep 16 12:55:49.555926 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:49.555887    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9742c8d6-9162-460b-8426-ecbde47a077a-dshm\") pod \"simulator-w0-490797-kserve-7fb7c79cf5-w9scf\" (UID: \"9742c8d6-9162-460b-8426-ecbde47a077a\") " pod="e2e-models-e2e-worker-w0-490797/simulator-w0-490797-kserve-7fb7c79cf5-w9scf"
Sep 16 12:55:49.555926 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:49.555929    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9742c8d6-9162-460b-8426-ecbde47a077a-tmp-dir\") pod \"simulator-w0-490797-kserve-7fb7c79cf5-w9scf\" (UID: \"9742c8d6-9162-460b-8426-ecbde47a077a\") " pod="e2e-models-e2e-worker-w0-490797/simulator-w0-490797-kserve-7fb7c79cf5-w9scf"
Sep 16 12:55:49.556159 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:49.556002    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9742c8d6-9162-460b-8426-ecbde47a077a-model-cache\") pod \"simulator-w0-490797-kserve-7fb7c79cf5-w9scf\" (UID: \"9742c8d6-9162-460b-8426-ecbde47a077a\") " pod="e2e-models-e2e-worker-w0-490797/simulator-w0-490797-kserve-7fb7c79cf5-w9scf"
Sep 16 12:55:49.556159 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:49.556084    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9742c8d6-9162-460b-8426-ecbde47a077a-home\") pod \"simulator-w0-490797-kserve-7fb7c79cf5-w9scf\" (UID: \"9742c8d6-9162-460b-8426-ecbde47a077a\") " pod="e2e-models-e2e-worker-w0-490797/simulator-w0-490797-kserve-7fb7c79cf5-w9scf"
Sep 16 12:55:49.556159 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:49.556121    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9742c8d6-9162-460b-8426-ecbde47a077a-tls-certs\") pod \"simulator-w0-490797-kserve-7fb7c79cf5-w9scf\" (UID: \"9742c8d6-9162-460b-8426-ecbde47a077a\") " pod="e2e-models-e2e-worker-w0-490797/simulator-w0-490797-kserve-7fb7c79cf5-w9scf"
Sep 16 12:55:49.556274 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:49.556176    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8vw7\" (UniqueName: \"kubernetes.io/projected/9742c8d6-9162-460b-8426-ecbde47a077a-kube-api-access-b8vw7\") pod \"simulator-w0-490797-kserve-7fb7c79cf5-w9scf\" (UID: \"9742c8d6-9162-460b-8426-ecbde47a077a\") " pod="e2e-models-e2e-worker-w0-490797/simulator-w0-490797-kserve-7fb7c79cf5-w9scf"
Sep 16 12:55:49.637998 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:49.637965    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"
Sep 16 12:55:49.638197 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:49.638176    2577 patch_prober.go:28] interesting pod/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 16 12:55:49.638268 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:49.638227    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp" podUID="7ad95431-ff85-4e77-9e49-c222239e9bce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 16 12:55:49.657647 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:49.657622    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9742c8d6-9162-460b-8426-ecbde47a077a-dshm\") pod \"simulator-w0-490797-kserve-7fb7c79cf5-w9scf\" (UID: \"9742c8d6-9162-460b-8426-ecbde47a077a\") " pod="e2e-models-e2e-worker-w0-490797/simulator-w0-490797-kserve-7fb7c79cf5-w9scf"
Sep 16 12:55:49.657762 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:49.657652    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9742c8d6-9162-460b-8426-ecbde47a077a-tmp-dir\") pod \"simulator-w0-490797-kserve-7fb7c79cf5-w9scf\" (UID: \"9742c8d6-9162-460b-8426-ecbde47a077a\") " pod="e2e-models-e2e-worker-w0-490797/simulator-w0-490797-kserve-7fb7c79cf5-w9scf"
Sep 16 12:55:49.657762 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:49.657693    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9742c8d6-9162-460b-8426-ecbde47a077a-model-cache\") pod \"simulator-w0-490797-kserve-7fb7c79cf5-w9scf\" (UID: \"9742c8d6-9162-460b-8426-ecbde47a077a\") " pod="e2e-models-e2e-worker-w0-490797/simulator-w0-490797-kserve-7fb7c79cf5-w9scf"
Sep 16 12:55:49.657762 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:49.657728    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9742c8d6-9162-460b-8426-ecbde47a077a-home\") pod \"simulator-w0-490797-kserve-7fb7c79cf5-w9scf\" (UID: \"9742c8d6-9162-460b-8426-ecbde47a077a\") " pod="e2e-models-e2e-worker-w0-490797/simulator-w0-490797-kserve-7fb7c79cf5-w9scf"
Sep 16 12:55:49.657762 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:49.657743    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9742c8d6-9162-460b-8426-ecbde47a077a-tls-certs\") pod \"simulator-w0-490797-kserve-7fb7c79cf5-w9scf\" (UID: \"9742c8d6-9162-460b-8426-ecbde47a077a\") " pod="e2e-models-e2e-worker-w0-490797/simulator-w0-490797-kserve-7fb7c79cf5-w9scf"
Sep 16 12:55:49.657961 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:49.657762    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8vw7\" (UniqueName: \"kubernetes.io/projected/9742c8d6-9162-460b-8426-ecbde47a077a-kube-api-access-b8vw7\") pod \"simulator-w0-490797-kserve-7fb7c79cf5-w9scf\" (UID: \"9742c8d6-9162-460b-8426-ecbde47a077a\") " pod="e2e-models-e2e-worker-w0-490797/simulator-w0-490797-kserve-7fb7c79cf5-w9scf"
Sep 16 12:55:49.658096 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:49.658064    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9742c8d6-9162-460b-8426-ecbde47a077a-tmp-dir\") pod \"simulator-w0-490797-kserve-7fb7c79cf5-w9scf\" (UID: \"9742c8d6-9162-460b-8426-ecbde47a077a\") " pod="e2e-models-e2e-worker-w0-490797/simulator-w0-490797-kserve-7fb7c79cf5-w9scf"
Sep 16 12:55:49.658222 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:49.658153    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9742c8d6-9162-460b-8426-ecbde47a077a-home\") pod \"simulator-w0-490797-kserve-7fb7c79cf5-w9scf\" (UID: \"9742c8d6-9162-460b-8426-ecbde47a077a\") " pod="e2e-models-e2e-worker-w0-490797/simulator-w0-490797-kserve-7fb7c79cf5-w9scf"
Sep 16 12:55:49.658222 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:49.658187    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9742c8d6-9162-460b-8426-ecbde47a077a-model-cache\") pod \"simulator-w0-490797-kserve-7fb7c79cf5-w9scf\" (UID: \"9742c8d6-9162-460b-8426-ecbde47a077a\") " pod="e2e-models-e2e-worker-w0-490797/simulator-w0-490797-kserve-7fb7c79cf5-w9scf"
Sep 16 12:55:49.660066 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:49.660042    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9742c8d6-9162-460b-8426-ecbde47a077a-dshm\") pod \"simulator-w0-490797-kserve-7fb7c79cf5-w9scf\" (UID: \"9742c8d6-9162-460b-8426-ecbde47a077a\") " pod="e2e-models-e2e-worker-w0-490797/simulator-w0-490797-kserve-7fb7c79cf5-w9scf"
Sep 16 12:55:49.660421 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:49.660402    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9742c8d6-9162-460b-8426-ecbde47a077a-tls-certs\") pod \"simulator-w0-490797-kserve-7fb7c79cf5-w9scf\" (UID: \"9742c8d6-9162-460b-8426-ecbde47a077a\") " pod="e2e-models-e2e-worker-w0-490797/simulator-w0-490797-kserve-7fb7c79cf5-w9scf"
Sep 16 12:55:49.666848 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:49.666822    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8vw7\" (UniqueName: \"kubernetes.io/projected/9742c8d6-9162-460b-8426-ecbde47a077a-kube-api-access-b8vw7\") pod \"simulator-w0-490797-kserve-7fb7c79cf5-w9scf\" (UID: \"9742c8d6-9162-460b-8426-ecbde47a077a\") " pod="e2e-models-e2e-worker-w0-490797/simulator-w0-490797-kserve-7fb7c79cf5-w9scf"
Sep 16 12:55:49.781119 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:49.781089    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-490797/simulator-w0-490797-kserve-7fb7c79cf5-w9scf"
Sep 16 12:55:50.080757 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:50.080734    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-490797/simulator-w0-490797-kserve-7fb7c79cf5-w9scf"]
Sep 16 12:55:50.082883 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:55:50.082859    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9742c8d6_9162_460b_8426_ecbde47a077a.slice/crio-95d0a7709404c53ca21b511a73aa8568c7aa9880403fdebd1db742ab2fccab82 WatchSource:0}: Error finding container 95d0a7709404c53ca21b511a73aa8568c7aa9880403fdebd1db742ab2fccab82: Status 404 returned error can't find the container with id 95d0a7709404c53ca21b511a73aa8568c7aa9880403fdebd1db742ab2fccab82
Sep 16 12:55:50.639215 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:50.639179    2577 patch_prober.go:28] interesting pod/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 16 12:55:50.639388 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:50.639248    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp" podUID="7ad95431-ff85-4e77-9e49-c222239e9bce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 16 12:55:51.036527 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:51.036472    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-490797/simulator-w0-490797-kserve-7fb7c79cf5-w9scf" event={"ID":"9742c8d6-9162-460b-8426-ecbde47a077a","Type":"ContainerStarted","Data":"d1b95dfed751cf34f4aaf52368d8a24fab172cd4c48f2870f45ddbdf2ed40008"}
Sep 16 12:55:51.036707 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:51.036541    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-490797/simulator-w0-490797-kserve-7fb7c79cf5-w9scf" event={"ID":"9742c8d6-9162-460b-8426-ecbde47a077a","Type":"ContainerStarted","Data":"95d0a7709404c53ca21b511a73aa8568c7aa9880403fdebd1db742ab2fccab82"}
Sep 16 12:55:51.056706 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:51.056656    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-490797/simulator-w0-490797-kserve-7fb7c79cf5-w9scf" podStartSLOduration=2.056640309 podStartE2EDuration="2.056640309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:55:51.054409762 +0000 UTC m=+1057.501642403" watchObservedRunningTime="2026-09-16 12:55:51.056640309 +0000 UTC m=+1057.503872991"
Sep 16 12:55:51.639085 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:51.639020    2577 patch_prober.go:28] interesting pod/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 16 12:55:51.639469 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:51.639096    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp" podUID="7ad95431-ff85-4e77-9e49-c222239e9bce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 16 12:55:52.638506 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:52.638473    2577 patch_prober.go:28] interesting pod/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 16 12:55:52.638685 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:52.638615    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp" podUID="7ad95431-ff85-4e77-9e49-c222239e9bce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 16 12:55:53.639191 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:53.639156    2577 patch_prober.go:28] interesting pod/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 16 12:55:53.639581 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:53.639218    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp" podUID="7ad95431-ff85-4e77-9e49-c222239e9bce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 16 12:55:54.641944 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:54.641914    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"
Sep 16 12:55:54.642416 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:54.642323    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"
Sep 16 12:55:54.643006 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:54.642988    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"
Sep 16 12:55:56.501484 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:56.501445    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e3535999-xrgqb"]
Sep 16 12:55:56.505476 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:56.505455    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e3535999-xrgqb"
Sep 16 12:55:56.508401 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:56.508374    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-64zph\""
Sep 16 12:55:56.508503 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:56.508375    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:55:56.516257 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:56.516234    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e3535999-xrgqb"]
Sep 16 12:55:56.635240 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:56.635194    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/481e9f6d-96f4-45ff-9819-9b867e5eabbe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e3535999-xrgqb\" (UID: \"481e9f6d-96f4-45ff-9819-9b867e5eabbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e3535999-xrgqb"
Sep 16 12:55:56.736173 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:56.736142    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/481e9f6d-96f4-45ff-9819-9b867e5eabbe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e3535999-xrgqb\" (UID: \"481e9f6d-96f4-45ff-9819-9b867e5eabbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e3535999-xrgqb"
Sep 16 12:55:56.736766 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:56.736746    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/481e9f6d-96f4-45ff-9819-9b867e5eabbe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e3535999-xrgqb\" (UID: \"481e9f6d-96f4-45ff-9819-9b867e5eabbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e3535999-xrgqb"
Sep 16 12:55:56.817980 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:56.817947    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e3535999-xrgqb"
Sep 16 12:55:57.095682 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:57.095508    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e3535999-xrgqb"]
Sep 16 12:55:57.098548 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:55:57.098516    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod481e9f6d_96f4_45ff_9819_9b867e5eabbe.slice/crio-66e698d59930bb954757ad9489bf8cedb2ef31c04144be9f2be25eb15c640951 WatchSource:0}: Error finding container 66e698d59930bb954757ad9489bf8cedb2ef31c04144be9f2be25eb15c640951: Status 404 returned error can't find the container with id 66e698d59930bb954757ad9489bf8cedb2ef31c04144be9f2be25eb15c640951
Sep 16 12:55:58.078578 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:58.078544    2577 generic.go:356] "Generic (PLEG): container finished" podID="481e9f6d-96f4-45ff-9819-9b867e5eabbe" containerID="1a66bb79e44e4a03ffaffdd1bb6463b7f5e83f72663c7138be70f2226eb1622f" exitCode=0
Sep 16 12:55:58.078938 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:58.078635    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e3535999-xrgqb" event={"ID":"481e9f6d-96f4-45ff-9819-9b867e5eabbe","Type":"ContainerDied","Data":"1a66bb79e44e4a03ffaffdd1bb6463b7f5e83f72663c7138be70f2226eb1622f"}
Sep 16 12:55:58.078938 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:58.078695    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e3535999-xrgqb" event={"ID":"481e9f6d-96f4-45ff-9819-9b867e5eabbe","Type":"ContainerStarted","Data":"66e698d59930bb954757ad9489bf8cedb2ef31c04144be9f2be25eb15c640951"}
Sep 16 12:55:59.236930 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:59.236899    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e3535999-xrgqb"
Sep 16 12:55:59.255261 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:59.255230    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/481e9f6d-96f4-45ff-9819-9b867e5eabbe-maas-api-service-ca\") pod \"481e9f6d-96f4-45ff-9819-9b867e5eabbe\" (UID: \"481e9f6d-96f4-45ff-9819-9b867e5eabbe\") "
Sep 16 12:55:59.255788 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:59.255745    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/481e9f6d-96f4-45ff-9819-9b867e5eabbe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "481e9f6d-96f4-45ff-9819-9b867e5eabbe" (UID: "481e9f6d-96f4-45ff-9819-9b867e5eabbe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:55:59.356777 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:59.356698    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/481e9f6d-96f4-45ff-9819-9b867e5eabbe-maas-api-service-ca\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:55:59.781411 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:59.781376    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-490797/simulator-w0-490797-kserve-7fb7c79cf5-w9scf"
Sep 16 12:55:59.781609 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:59.781424    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-490797/simulator-w0-490797-kserve-7fb7c79cf5-w9scf"
Sep 16 12:55:59.794872 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:55:59.794851    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-490797/simulator-w0-490797-kserve-7fb7c79cf5-w9scf"
Sep 16 12:56:00.093842 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:00.093815    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e3535999-xrgqb"
Sep 16 12:56:00.094001 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:00.093814    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e3535999-xrgqb" event={"ID":"481e9f6d-96f4-45ff-9819-9b867e5eabbe","Type":"ContainerDied","Data":"66e698d59930bb954757ad9489bf8cedb2ef31c04144be9f2be25eb15c640951"}
Sep 16 12:56:00.094001 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:00.093952    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66e698d59930bb954757ad9489bf8cedb2ef31c04144be9f2be25eb15c640951"
Sep 16 12:56:00.105785 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:00.105762    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-490797/simulator-w0-490797-kserve-7fb7c79cf5-w9scf"
Sep 16 12:56:08.227144 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:08.227058    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e3535999-bb8778658-f8p24"]
Sep 16 12:56:08.227578 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:08.227341    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e3535999-bb8778658-f8p24" podUID="9606b33d-d777-436c-87f0-4b9548132a60" containerName="payload-pre-processing" containerID="cri-o://0387eadd973a4608a78924aa10e2a29c820d92512a7006ece74bc559d4e6b8d2" gracePeriod=30
Sep 16 12:56:08.385525 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:08.385484    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e3535999-xrgqb"]
Sep 16 12:56:08.387694 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:08.387665    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e3535999-xrgqb"]
Sep 16 12:56:08.526569 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:08.526546    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e3535999-bb8778658-f8p24"
Sep 16 12:56:08.529412 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:08.529383    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"]
Sep 16 12:56:08.529924 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:08.529673    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp" podUID="7ad95431-ff85-4e77-9e49-c222239e9bce" containerName="istio-proxy" containerID="cri-o://21b42f9f58d03f45b00f2f36625f3479ca9fed10ec9f676060e4feaf541fc33e" gracePeriod=30
Sep 16 12:56:08.652177 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:08.651545    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcl8r\" (UniqueName: \"kubernetes.io/projected/9606b33d-d777-436c-87f0-4b9548132a60-kube-api-access-xcl8r\") pod \"9606b33d-d777-436c-87f0-4b9548132a60\" (UID: \"9606b33d-d777-436c-87f0-4b9548132a60\") "
Sep 16 12:56:08.652177 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:08.651600    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9606b33d-d777-436c-87f0-4b9548132a60-plugins-config-volume\") pod \"9606b33d-d777-436c-87f0-4b9548132a60\" (UID: \"9606b33d-d777-436c-87f0-4b9548132a60\") "
Sep 16 12:56:08.652594 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:08.652554    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9606b33d-d777-436c-87f0-4b9548132a60-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9606b33d-d777-436c-87f0-4b9548132a60" (UID: "9606b33d-d777-436c-87f0-4b9548132a60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:56:08.658430 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:08.658394    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9606b33d-d777-436c-87f0-4b9548132a60-kube-api-access-xcl8r" (OuterVolumeSpecName: "kube-api-access-xcl8r") pod "9606b33d-d777-436c-87f0-4b9548132a60" (UID: "9606b33d-d777-436c-87f0-4b9548132a60"). InnerVolumeSpecName "kube-api-access-xcl8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:56:08.752914 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:08.752839    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcl8r\" (UniqueName: \"kubernetes.io/projected/9606b33d-d777-436c-87f0-4b9548132a60-kube-api-access-xcl8r\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:56:08.752914 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:08.752865    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9606b33d-d777-436c-87f0-4b9548132a60-plugins-config-volume\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:56:09.149530 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.149498    2577 generic.go:356] "Generic (PLEG): container finished" podID="9606b33d-d777-436c-87f0-4b9548132a60" containerID="0387eadd973a4608a78924aa10e2a29c820d92512a7006ece74bc559d4e6b8d2" exitCode=0
Sep 16 12:56:09.149730 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.149548    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e3535999-bb8778658-f8p24"
Sep 16 12:56:09.149730 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.149582    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e3535999-bb8778658-f8p24" event={"ID":"9606b33d-d777-436c-87f0-4b9548132a60","Type":"ContainerDied","Data":"0387eadd973a4608a78924aa10e2a29c820d92512a7006ece74bc559d4e6b8d2"}
Sep 16 12:56:09.149730 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.149636    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e3535999-bb8778658-f8p24" event={"ID":"9606b33d-d777-436c-87f0-4b9548132a60","Type":"ContainerDied","Data":"c1fb2ddcfbe39cd0358cf8ed08e8bfb0deec1177f0ecb67ecc5c5afd0a10279e"}
Sep 16 12:56:09.149730 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.149657    2577 scope.go:117] "RemoveContainer" containerID="0387eadd973a4608a78924aa10e2a29c820d92512a7006ece74bc559d4e6b8d2"
Sep 16 12:56:09.159763 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.159748    2577 scope.go:117] "RemoveContainer" containerID="0387eadd973a4608a78924aa10e2a29c820d92512a7006ece74bc559d4e6b8d2"
Sep 16 12:56:09.160011 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:56:09.159992    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0387eadd973a4608a78924aa10e2a29c820d92512a7006ece74bc559d4e6b8d2\": container with ID starting with 0387eadd973a4608a78924aa10e2a29c820d92512a7006ece74bc559d4e6b8d2 not found: ID does not exist" containerID="0387eadd973a4608a78924aa10e2a29c820d92512a7006ece74bc559d4e6b8d2"
Sep 16 12:56:09.160112 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.160016    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0387eadd973a4608a78924aa10e2a29c820d92512a7006ece74bc559d4e6b8d2"} err="failed to get container status \"0387eadd973a4608a78924aa10e2a29c820d92512a7006ece74bc559d4e6b8d2\": rpc error: code = NotFound desc = could not find container \"0387eadd973a4608a78924aa10e2a29c820d92512a7006ece74bc559d4e6b8d2\": container with ID starting with 0387eadd973a4608a78924aa10e2a29c820d92512a7006ece74bc559d4e6b8d2 not found: ID does not exist"
Sep 16 12:56:09.174337 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.174312    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e3535999-bb8778658-f8p24"]
Sep 16 12:56:09.180276 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.180256    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e3535999-bb8778658-f8p24"]
Sep 16 12:56:09.282677 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.282646    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"]
Sep 16 12:56:09.283188 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.283173    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="481e9f6d-96f4-45ff-9819-9b867e5eabbe" containerName="revoke-keys"
Sep 16 12:56:09.283257 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.283190    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="481e9f6d-96f4-45ff-9819-9b867e5eabbe" containerName="revoke-keys"
Sep 16 12:56:09.283257 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.283206    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9606b33d-d777-436c-87f0-4b9548132a60" containerName="payload-pre-processing"
Sep 16 12:56:09.283257 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.283214    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9606b33d-d777-436c-87f0-4b9548132a60" containerName="payload-pre-processing"
Sep 16 12:56:09.283364 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.283309    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="481e9f6d-96f4-45ff-9819-9b867e5eabbe" containerName="revoke-keys"
Sep 16 12:56:09.283364 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.283324    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9606b33d-d777-436c-87f0-4b9548132a60" containerName="payload-pre-processing"
Sep 16 12:56:09.287940 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.287916    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"
Sep 16 12:56:09.290500 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.290475    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-19dd11cf-openshift-default-dockercfg-dddzw\""
Sep 16 12:56:09.298387 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.298363    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"]
Sep 16 12:56:09.357387 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.357360    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cba4736f-ac79-494d-95db-4cf94129d658-istio-envoy\") pod \"e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9\" (UID: \"cba4736f-ac79-494d-95db-4cf94129d658\") " pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"
Sep 16 12:56:09.357537 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.357394    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frcvv\" (UniqueName: \"kubernetes.io/projected/cba4736f-ac79-494d-95db-4cf94129d658-kube-api-access-frcvv\") pod \"e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9\" (UID: \"cba4736f-ac79-494d-95db-4cf94129d658\") " pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"
Sep 16 12:56:09.357537 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.357433    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cba4736f-ac79-494d-95db-4cf94129d658-istio-podinfo\") pod \"e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9\" (UID: \"cba4736f-ac79-494d-95db-4cf94129d658\") " pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"
Sep 16 12:56:09.357537 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.357519    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cba4736f-ac79-494d-95db-4cf94129d658-workload-certs\") pod \"e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9\" (UID: \"cba4736f-ac79-494d-95db-4cf94129d658\") " pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"
Sep 16 12:56:09.357657 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.357557    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba4736f-ac79-494d-95db-4cf94129d658-workload-socket\") pod \"e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9\" (UID: \"cba4736f-ac79-494d-95db-4cf94129d658\") " pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"
Sep 16 12:56:09.357657 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.357572    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cba4736f-ac79-494d-95db-4cf94129d658-istio-token\") pod \"e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9\" (UID: \"cba4736f-ac79-494d-95db-4cf94129d658\") " pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"
Sep 16 12:56:09.357657 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.357641    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba4736f-ac79-494d-95db-4cf94129d658-credential-socket\") pod \"e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9\" (UID: \"cba4736f-ac79-494d-95db-4cf94129d658\") " pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"
Sep 16 12:56:09.357767 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.357703    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cba4736f-ac79-494d-95db-4cf94129d658-istio-data\") pod \"e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9\" (UID: \"cba4736f-ac79-494d-95db-4cf94129d658\") " pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"
Sep 16 12:56:09.357767 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.357724    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cba4736f-ac79-494d-95db-4cf94129d658-istiod-ca-cert\") pod \"e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9\" (UID: \"cba4736f-ac79-494d-95db-4cf94129d658\") " pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"
Sep 16 12:56:09.458730 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.458643    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cba4736f-ac79-494d-95db-4cf94129d658-istio-data\") pod \"e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9\" (UID: \"cba4736f-ac79-494d-95db-4cf94129d658\") " pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"
Sep 16 12:56:09.458730 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.458682    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cba4736f-ac79-494d-95db-4cf94129d658-istiod-ca-cert\") pod \"e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9\" (UID: \"cba4736f-ac79-494d-95db-4cf94129d658\") " pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"
Sep 16 12:56:09.458730 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.458700    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cba4736f-ac79-494d-95db-4cf94129d658-istio-envoy\") pod \"e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9\" (UID: \"cba4736f-ac79-494d-95db-4cf94129d658\") " pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"
Sep 16 12:56:09.459015 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.458825    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frcvv\" (UniqueName: \"kubernetes.io/projected/cba4736f-ac79-494d-95db-4cf94129d658-kube-api-access-frcvv\") pod \"e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9\" (UID: \"cba4736f-ac79-494d-95db-4cf94129d658\") " pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"
Sep 16 12:56:09.459015 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.458901    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cba4736f-ac79-494d-95db-4cf94129d658-istio-podinfo\") pod \"e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9\" (UID: \"cba4736f-ac79-494d-95db-4cf94129d658\") " pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"
Sep 16 12:56:09.459015 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.458980    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cba4736f-ac79-494d-95db-4cf94129d658-workload-certs\") pod \"e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9\" (UID: \"cba4736f-ac79-494d-95db-4cf94129d658\") " pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"
Sep 16 12:56:09.459230 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.459020    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba4736f-ac79-494d-95db-4cf94129d658-workload-socket\") pod \"e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9\" (UID: \"cba4736f-ac79-494d-95db-4cf94129d658\") " pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"
Sep 16 12:56:09.459230 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.459015    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cba4736f-ac79-494d-95db-4cf94129d658-istio-data\") pod \"e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9\" (UID: \"cba4736f-ac79-494d-95db-4cf94129d658\") " pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"
Sep 16 12:56:09.459230 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.459062    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cba4736f-ac79-494d-95db-4cf94129d658-istio-token\") pod \"e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9\" (UID: \"cba4736f-ac79-494d-95db-4cf94129d658\") " pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"
Sep 16 12:56:09.459505 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.459485    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba4736f-ac79-494d-95db-4cf94129d658-credential-socket\") pod \"e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9\" (UID: \"cba4736f-ac79-494d-95db-4cf94129d658\") " pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"
Sep 16 12:56:09.459582 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.459333    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cba4736f-ac79-494d-95db-4cf94129d658-workload-certs\") pod \"e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9\" (UID: \"cba4736f-ac79-494d-95db-4cf94129d658\") " pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"
Sep 16 12:56:09.459639 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.459420    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cba4736f-ac79-494d-95db-4cf94129d658-istiod-ca-cert\") pod \"e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9\" (UID: \"cba4736f-ac79-494d-95db-4cf94129d658\") " pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"
Sep 16 12:56:09.459691 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.459412    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba4736f-ac79-494d-95db-4cf94129d658-workload-socket\") pod \"e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9\" (UID: \"cba4736f-ac79-494d-95db-4cf94129d658\") " pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"
Sep 16 12:56:09.459936 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.459915    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba4736f-ac79-494d-95db-4cf94129d658-credential-socket\") pod \"e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9\" (UID: \"cba4736f-ac79-494d-95db-4cf94129d658\") " pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"
Sep 16 12:56:09.461234 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.461213    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cba4736f-ac79-494d-95db-4cf94129d658-istio-envoy\") pod \"e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9\" (UID: \"cba4736f-ac79-494d-95db-4cf94129d658\") " pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"
Sep 16 12:56:09.461489 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.461473    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cba4736f-ac79-494d-95db-4cf94129d658-istio-podinfo\") pod \"e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9\" (UID: \"cba4736f-ac79-494d-95db-4cf94129d658\") " pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"
Sep 16 12:56:09.467124 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.467097    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cba4736f-ac79-494d-95db-4cf94129d658-istio-token\") pod \"e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9\" (UID: \"cba4736f-ac79-494d-95db-4cf94129d658\") " pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"
Sep 16 12:56:09.467330 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.467313    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frcvv\" (UniqueName: \"kubernetes.io/projected/cba4736f-ac79-494d-95db-4cf94129d658-kube-api-access-frcvv\") pod \"e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9\" (UID: \"cba4736f-ac79-494d-95db-4cf94129d658\") " pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"
Sep 16 12:56:09.601432 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.601399    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"
Sep 16 12:56:09.644692 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.644656    2577 patch_prober.go:28] interesting pod/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 12:56:09.644839 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.644717    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp" podUID="7ad95431-ff85-4e77-9e49-c222239e9bce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 12:56:09.893497 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.893471    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"]
Sep 16 12:56:09.894684 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:56:09.894661    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcba4736f_ac79_494d_95db_4cf94129d658.slice/crio-cf7c3fd938ede7214305fc232c978b92b6b1cdf4fb609f93923db70a1ae0dc50 WatchSource:0}: Error finding container cf7c3fd938ede7214305fc232c978b92b6b1cdf4fb609f93923db70a1ae0dc50: Status 404 returned error can't find the container with id cf7c3fd938ede7214305fc232c978b92b6b1cdf4fb609f93923db70a1ae0dc50
Sep 16 12:56:09.897710 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.897673    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:56:09.897818 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.897744    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:56:09.897818 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:09.897777    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:56:10.056720 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:10.056658    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="481e9f6d-96f4-45ff-9819-9b867e5eabbe" path="/var/lib/kubelet/pods/481e9f6d-96f4-45ff-9819-9b867e5eabbe/volumes"
Sep 16 12:56:10.057204 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:10.057188    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9606b33d-d777-436c-87f0-4b9548132a60" path="/var/lib/kubelet/pods/9606b33d-d777-436c-87f0-4b9548132a60/volumes"
Sep 16 12:56:10.158953 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:10.158847    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9" event={"ID":"cba4736f-ac79-494d-95db-4cf94129d658","Type":"ContainerStarted","Data":"808c9466723b1d04a9570a3663996c45da9a26df0efe1db595fa6cca010940e4"}
Sep 16 12:56:10.158953 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:10.158911    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9" event={"ID":"cba4736f-ac79-494d-95db-4cf94129d658","Type":"ContainerStarted","Data":"cf7c3fd938ede7214305fc232c978b92b6b1cdf4fb609f93923db70a1ae0dc50"}
Sep 16 12:56:10.194048 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:10.192377    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9" podStartSLOduration=1.192356274 podStartE2EDuration="1.192356274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:56:10.184592532 +0000 UTC m=+1076.631825175" watchObservedRunningTime="2026-09-16 12:56:10.192356274 +0000 UTC m=+1076.639588917"
Sep 16 12:56:10.602248 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:10.602215    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"
Sep 16 12:56:10.606796 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:10.606768    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"
Sep 16 12:56:11.166258 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:11.166226    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"
Sep 16 12:56:11.166986 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:11.166969    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"
Sep 16 12:56:13.789685 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:13.789659    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"
Sep 16 12:56:13.906975 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:13.906883    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ad95431-ff85-4e77-9e49-c222239e9bce-istio-envoy\") pod \"7ad95431-ff85-4e77-9e49-c222239e9bce\" (UID: \"7ad95431-ff85-4e77-9e49-c222239e9bce\") "
Sep 16 12:56:13.907157 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:13.906979    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ad95431-ff85-4e77-9e49-c222239e9bce-istio-podinfo\") pod \"7ad95431-ff85-4e77-9e49-c222239e9bce\" (UID: \"7ad95431-ff85-4e77-9e49-c222239e9bce\") "
Sep 16 12:56:13.907157 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:13.907050    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ad95431-ff85-4e77-9e49-c222239e9bce-istiod-ca-cert\") pod \"7ad95431-ff85-4e77-9e49-c222239e9bce\" (UID: \"7ad95431-ff85-4e77-9e49-c222239e9bce\") "
Sep 16 12:56:13.907284 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:13.907170    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad95431-ff85-4e77-9e49-c222239e9bce-credential-socket\") pod \"7ad95431-ff85-4e77-9e49-c222239e9bce\" (UID: \"7ad95431-ff85-4e77-9e49-c222239e9bce\") "
Sep 16 12:56:13.907284 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:13.907228    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ad95431-ff85-4e77-9e49-c222239e9bce-workload-certs\") pod \"7ad95431-ff85-4e77-9e49-c222239e9bce\" (UID: \"7ad95431-ff85-4e77-9e49-c222239e9bce\") "
Sep 16 12:56:13.907284 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:13.907260    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvx6l\" (UniqueName: \"kubernetes.io/projected/7ad95431-ff85-4e77-9e49-c222239e9bce-kube-api-access-gvx6l\") pod \"7ad95431-ff85-4e77-9e49-c222239e9bce\" (UID: \"7ad95431-ff85-4e77-9e49-c222239e9bce\") "
Sep 16 12:56:13.907435 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:13.907290    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ad95431-ff85-4e77-9e49-c222239e9bce-istio-token\") pod \"7ad95431-ff85-4e77-9e49-c222239e9bce\" (UID: \"7ad95431-ff85-4e77-9e49-c222239e9bce\") "
Sep 16 12:56:13.907435 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:13.907348    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ad95431-ff85-4e77-9e49-c222239e9bce-istio-data\") pod \"7ad95431-ff85-4e77-9e49-c222239e9bce\" (UID: \"7ad95431-ff85-4e77-9e49-c222239e9bce\") "
Sep 16 12:56:13.907435 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:13.907373    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad95431-ff85-4e77-9e49-c222239e9bce-workload-socket\") pod \"7ad95431-ff85-4e77-9e49-c222239e9bce\" (UID: \"7ad95431-ff85-4e77-9e49-c222239e9bce\") "
Sep 16 12:56:13.907435 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:13.907408    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad95431-ff85-4e77-9e49-c222239e9bce-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7ad95431-ff85-4e77-9e49-c222239e9bce" (UID: "7ad95431-ff85-4e77-9e49-c222239e9bce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:56:13.907638 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:13.907451    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad95431-ff85-4e77-9e49-c222239e9bce-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7ad95431-ff85-4e77-9e49-c222239e9bce" (UID: "7ad95431-ff85-4e77-9e49-c222239e9bce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:56:13.907638 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:13.907506    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ad95431-ff85-4e77-9e49-c222239e9bce-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7ad95431-ff85-4e77-9e49-c222239e9bce" (UID: "7ad95431-ff85-4e77-9e49-c222239e9bce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:56:13.907638 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:13.907623    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad95431-ff85-4e77-9e49-c222239e9bce-istio-data" (OuterVolumeSpecName: "istio-data") pod "7ad95431-ff85-4e77-9e49-c222239e9bce" (UID: "7ad95431-ff85-4e77-9e49-c222239e9bce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:56:13.907794 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:13.907705    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad95431-ff85-4e77-9e49-c222239e9bce-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7ad95431-ff85-4e77-9e49-c222239e9bce" (UID: "7ad95431-ff85-4e77-9e49-c222239e9bce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:56:13.907861 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:13.907845    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ad95431-ff85-4e77-9e49-c222239e9bce-workload-certs\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:56:13.908111 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:13.907862    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ad95431-ff85-4e77-9e49-c222239e9bce-istio-data\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:56:13.908111 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:13.907871    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad95431-ff85-4e77-9e49-c222239e9bce-workload-socket\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:56:13.908111 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:13.907879    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ad95431-ff85-4e77-9e49-c222239e9bce-istiod-ca-cert\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:56:13.908111 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:13.907887    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad95431-ff85-4e77-9e49-c222239e9bce-credential-socket\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:56:13.909538 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:13.909510    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad95431-ff85-4e77-9e49-c222239e9bce-kube-api-access-gvx6l" (OuterVolumeSpecName: "kube-api-access-gvx6l") pod "7ad95431-ff85-4e77-9e49-c222239e9bce" (UID: "7ad95431-ff85-4e77-9e49-c222239e9bce"). InnerVolumeSpecName "kube-api-access-gvx6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:56:13.909673 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:13.909646    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad95431-ff85-4e77-9e49-c222239e9bce-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7ad95431-ff85-4e77-9e49-c222239e9bce" (UID: "7ad95431-ff85-4e77-9e49-c222239e9bce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:56:13.909984 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:13.909958    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad95431-ff85-4e77-9e49-c222239e9bce-istio-token" (OuterVolumeSpecName: "istio-token") pod "7ad95431-ff85-4e77-9e49-c222239e9bce" (UID: "7ad95431-ff85-4e77-9e49-c222239e9bce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:56:13.909984 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:13.909958    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ad95431-ff85-4e77-9e49-c222239e9bce-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7ad95431-ff85-4e77-9e49-c222239e9bce" (UID: "7ad95431-ff85-4e77-9e49-c222239e9bce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:56:14.008415 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:14.008386    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ad95431-ff85-4e77-9e49-c222239e9bce-istio-envoy\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:56:14.008415 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:14.008413    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ad95431-ff85-4e77-9e49-c222239e9bce-istio-podinfo\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:56:14.008655 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:14.008427    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvx6l\" (UniqueName: \"kubernetes.io/projected/7ad95431-ff85-4e77-9e49-c222239e9bce-kube-api-access-gvx6l\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:56:14.008655 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:14.008441    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ad95431-ff85-4e77-9e49-c222239e9bce-istio-token\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:56:14.190384 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:14.190289    2577 generic.go:356] "Generic (PLEG): container finished" podID="7ad95431-ff85-4e77-9e49-c222239e9bce" containerID="21b42f9f58d03f45b00f2f36625f3479ca9fed10ec9f676060e4feaf541fc33e" exitCode=0
Sep 16 12:56:14.190531 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:14.190388    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"
Sep 16 12:56:14.190628 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:14.190593    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp" event={"ID":"7ad95431-ff85-4e77-9e49-c222239e9bce","Type":"ContainerDied","Data":"21b42f9f58d03f45b00f2f36625f3479ca9fed10ec9f676060e4feaf541fc33e"}
Sep 16 12:56:14.190748 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:14.190728    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp" event={"ID":"7ad95431-ff85-4e77-9e49-c222239e9bce","Type":"ContainerDied","Data":"c87382ecf5a047e267ec0df29516fdbe4ed48b0bfa0ac6c1e72d8a6ab6c544b9"}
Sep 16 12:56:14.190830 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:14.190780    2577 scope.go:117] "RemoveContainer" containerID="21b42f9f58d03f45b00f2f36625f3479ca9fed10ec9f676060e4feaf541fc33e"
Sep 16 12:56:14.202388 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:14.202368    2577 scope.go:117] "RemoveContainer" containerID="901d59daecf83099c60bd91e23cd8763f6a7aaf29ec2ec89056d5a76ee80dec7"
Sep 16 12:56:14.209933 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:14.209906    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"]
Sep 16 12:56:14.211783 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:14.211765    2577 scope.go:117] "RemoveContainer" containerID="21b42f9f58d03f45b00f2f36625f3479ca9fed10ec9f676060e4feaf541fc33e"
Sep 16 12:56:14.212077 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:56:14.212060    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21b42f9f58d03f45b00f2f36625f3479ca9fed10ec9f676060e4feaf541fc33e\": container with ID starting with 21b42f9f58d03f45b00f2f36625f3479ca9fed10ec9f676060e4feaf541fc33e not found: ID does not exist" containerID="21b42f9f58d03f45b00f2f36625f3479ca9fed10ec9f676060e4feaf541fc33e"
Sep 16 12:56:14.212129 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:14.212084    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21b42f9f58d03f45b00f2f36625f3479ca9fed10ec9f676060e4feaf541fc33e"} err="failed to get container status \"21b42f9f58d03f45b00f2f36625f3479ca9fed10ec9f676060e4feaf541fc33e\": rpc error: code = NotFound desc = could not find container \"21b42f9f58d03f45b00f2f36625f3479ca9fed10ec9f676060e4feaf541fc33e\": container with ID starting with 21b42f9f58d03f45b00f2f36625f3479ca9fed10ec9f676060e4feaf541fc33e not found: ID does not exist"
Sep 16 12:56:14.212129 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:14.212102    2577 scope.go:117] "RemoveContainer" containerID="901d59daecf83099c60bd91e23cd8763f6a7aaf29ec2ec89056d5a76ee80dec7"
Sep 16 12:56:14.212515 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:56:14.212468    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"901d59daecf83099c60bd91e23cd8763f6a7aaf29ec2ec89056d5a76ee80dec7\": container with ID starting with 901d59daecf83099c60bd91e23cd8763f6a7aaf29ec2ec89056d5a76ee80dec7 not found: ID does not exist" containerID="901d59daecf83099c60bd91e23cd8763f6a7aaf29ec2ec89056d5a76ee80dec7"
Sep 16 12:56:14.212662 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:14.212561    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"901d59daecf83099c60bd91e23cd8763f6a7aaf29ec2ec89056d5a76ee80dec7"} err="failed to get container status \"901d59daecf83099c60bd91e23cd8763f6a7aaf29ec2ec89056d5a76ee80dec7\": rpc error: code = NotFound desc = could not find container \"901d59daecf83099c60bd91e23cd8763f6a7aaf29ec2ec89056d5a76ee80dec7\": container with ID starting with 901d59daecf83099c60bd91e23cd8763f6a7aaf29ec2ec89056d5a76ee80dec7 not found: ID does not exist"
Sep 16 12:56:14.215638 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:14.215615    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-e3535999-gw-openshift-default-786745bcb-4vdgp"]
Sep 16 12:56:14.887394 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:14.887291    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-19dd11cf-84b6456898-zfxm2"]
Sep 16 12:56:14.887902 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:14.887881    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ad95431-ff85-4e77-9e49-c222239e9bce" containerName="istio-proxy"
Sep 16 12:56:14.887902 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:14.887903    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ad95431-ff85-4e77-9e49-c222239e9bce" containerName="istio-proxy"
Sep 16 12:56:14.888067 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:14.887920    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ad95431-ff85-4e77-9e49-c222239e9bce" containerName="istio-proxy"
Sep 16 12:56:14.888067 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:14.887926    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ad95431-ff85-4e77-9e49-c222239e9bce" containerName="istio-proxy"
Sep 16 12:56:14.888067 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:14.888003    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ad95431-ff85-4e77-9e49-c222239e9bce" containerName="istio-proxy"
Sep 16 12:56:14.888067 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:14.888011    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ad95431-ff85-4e77-9e49-c222239e9bce" containerName="istio-proxy"
Sep 16 12:56:14.893185 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:14.893165    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-19dd11cf-84b6456898-zfxm2"
Sep 16 12:56:14.896308 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:14.896284    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-19dd11cf\""
Sep 16 12:56:14.896537 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:14.896525    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-19dd11cf-dockercfg-5k7rx\""
Sep 16 12:56:14.905644 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:14.905622    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-19dd11cf-84b6456898-zfxm2"]
Sep 16 12:56:15.020078 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:15.020005    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9ph2\" (UniqueName: \"kubernetes.io/projected/1072d0fd-8c68-414f-a74a-44ac8414d6c1-kube-api-access-n9ph2\") pod \"payload-pre-processing-e2e-ait-19dd11cf-84b6456898-zfxm2\" (UID: \"1072d0fd-8c68-414f-a74a-44ac8414d6c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-19dd11cf-84b6456898-zfxm2"
Sep 16 12:56:15.020368 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:15.020322    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1072d0fd-8c68-414f-a74a-44ac8414d6c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-19dd11cf-84b6456898-zfxm2\" (UID: \"1072d0fd-8c68-414f-a74a-44ac8414d6c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-19dd11cf-84b6456898-zfxm2"
Sep 16 12:56:15.121418 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:15.121381    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1072d0fd-8c68-414f-a74a-44ac8414d6c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-19dd11cf-84b6456898-zfxm2\" (UID: \"1072d0fd-8c68-414f-a74a-44ac8414d6c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-19dd11cf-84b6456898-zfxm2"
Sep 16 12:56:15.121593 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:15.121477    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9ph2\" (UniqueName: \"kubernetes.io/projected/1072d0fd-8c68-414f-a74a-44ac8414d6c1-kube-api-access-n9ph2\") pod \"payload-pre-processing-e2e-ait-19dd11cf-84b6456898-zfxm2\" (UID: \"1072d0fd-8c68-414f-a74a-44ac8414d6c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-19dd11cf-84b6456898-zfxm2"
Sep 16 12:56:15.122110 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:15.122087    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1072d0fd-8c68-414f-a74a-44ac8414d6c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-19dd11cf-84b6456898-zfxm2\" (UID: \"1072d0fd-8c68-414f-a74a-44ac8414d6c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-19dd11cf-84b6456898-zfxm2"
Sep 16 12:56:15.130322 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:15.130301    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9ph2\" (UniqueName: \"kubernetes.io/projected/1072d0fd-8c68-414f-a74a-44ac8414d6c1-kube-api-access-n9ph2\") pod \"payload-pre-processing-e2e-ait-19dd11cf-84b6456898-zfxm2\" (UID: \"1072d0fd-8c68-414f-a74a-44ac8414d6c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-19dd11cf-84b6456898-zfxm2"
Sep 16 12:56:15.205827 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:15.205754    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-19dd11cf-84b6456898-zfxm2"
Sep 16 12:56:15.479646 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:15.479538    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-19dd11cf-84b6456898-zfxm2"]
Sep 16 12:56:15.482251 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:56:15.482219    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1072d0fd_8c68_414f_a74a_44ac8414d6c1.slice/crio-0e67626b12a6213b9bea02729a3faa3a4b4bbee201268db6316736cc80a9f5d5 WatchSource:0}: Error finding container 0e67626b12a6213b9bea02729a3faa3a4b4bbee201268db6316736cc80a9f5d5: Status 404 returned error can't find the container with id 0e67626b12a6213b9bea02729a3faa3a4b4bbee201268db6316736cc80a9f5d5
Sep 16 12:56:16.055401 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:16.055370    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ad95431-ff85-4e77-9e49-c222239e9bce" path="/var/lib/kubelet/pods/7ad95431-ff85-4e77-9e49-c222239e9bce/volumes"
Sep 16 12:56:16.204766 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:16.204731    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-19dd11cf-84b6456898-zfxm2" event={"ID":"1072d0fd-8c68-414f-a74a-44ac8414d6c1","Type":"ContainerStarted","Data":"5a631c1accd84be189372a503dbfbe986e1297534a2d2055933b67077527440a"}
Sep 16 12:56:16.204967 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:16.204789    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-19dd11cf-84b6456898-zfxm2" event={"ID":"1072d0fd-8c68-414f-a74a-44ac8414d6c1","Type":"ContainerStarted","Data":"0e67626b12a6213b9bea02729a3faa3a4b4bbee201268db6316736cc80a9f5d5"}
Sep 16 12:56:16.204967 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:16.204857    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-19dd11cf-84b6456898-zfxm2"
Sep 16 12:56:16.222108 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:16.222068    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-19dd11cf-84b6456898-zfxm2" podStartSLOduration=2.222052142 podStartE2EDuration="2.222052142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:56:16.221337449 +0000 UTC m=+1082.668570093" watchObservedRunningTime="2026-09-16 12:56:16.222052142 +0000 UTC m=+1082.669284778"
Sep 16 12:56:27.214244 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:27.214211    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-19dd11cf-84b6456898-zfxm2"
Sep 16 12:56:34.316874 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:34.316843    2577 generic.go:356] "Generic (PLEG): container finished" podID="cba4736f-ac79-494d-95db-4cf94129d658" containerID="808c9466723b1d04a9570a3663996c45da9a26df0efe1db595fa6cca010940e4" exitCode=0
Sep 16 12:56:34.317436 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:34.316923    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9" event={"ID":"cba4736f-ac79-494d-95db-4cf94129d658","Type":"ContainerDied","Data":"808c9466723b1d04a9570a3663996c45da9a26df0efe1db595fa6cca010940e4"}
Sep 16 12:56:34.317524 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:34.317506    2577 scope.go:117] "RemoveContainer" containerID="808c9466723b1d04a9570a3663996c45da9a26df0efe1db595fa6cca010940e4"
Sep 16 12:56:34.318725 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:34.318700    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:56:34.318786 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:34.318764    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:56:34.318822 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:34.318795    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:56:35.324980 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:35.324947    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9" event={"ID":"cba4736f-ac79-494d-95db-4cf94129d658","Type":"ContainerStarted","Data":"b5e38ae32feea2958122f1b68123bc268400482fb27cdde219538c1ffabd0f81"}
Sep 16 12:56:35.601714 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:35.601612    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"
Sep 16 12:56:35.602365 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:35.602338    2577 patch_prober.go:28] interesting pod/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 16 12:56:35.602591 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:35.602564    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9" podUID="cba4736f-ac79-494d-95db-4cf94129d658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 16 12:56:36.602494 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:36.602437    2577 patch_prober.go:28] interesting pod/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 16 12:56:36.602984 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:36.602517    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9" podUID="cba4736f-ac79-494d-95db-4cf94129d658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 16 12:56:37.328556 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:37.328520    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19dd11cf-pnqtg"]
Sep 16 12:56:37.332528 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:37.332509    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19dd11cf-pnqtg"
Sep 16 12:56:37.335276 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:37.335255    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-64zph\""
Sep 16 12:56:37.335379 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:37.335256    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:56:37.342873 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:37.342848    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19dd11cf-pnqtg"]
Sep 16 12:56:37.347214 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:37.347184    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08d797af-03c0-475a-b2ca-4838df8ac6b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-19dd11cf-pnqtg\" (UID: \"08d797af-03c0-475a-b2ca-4838df8ac6b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19dd11cf-pnqtg"
Sep 16 12:56:37.448831 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:37.448801    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08d797af-03c0-475a-b2ca-4838df8ac6b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-19dd11cf-pnqtg\" (UID: \"08d797af-03c0-475a-b2ca-4838df8ac6b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19dd11cf-pnqtg"
Sep 16 12:56:37.449459 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:37.449438    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08d797af-03c0-475a-b2ca-4838df8ac6b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-19dd11cf-pnqtg\" (UID: \"08d797af-03c0-475a-b2ca-4838df8ac6b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19dd11cf-pnqtg"
Sep 16 12:56:37.602787 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:37.602690    2577 patch_prober.go:28] interesting pod/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 16 12:56:37.603358 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:37.603323    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9" podUID="cba4736f-ac79-494d-95db-4cf94129d658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 16 12:56:37.645691 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:37.645659    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19dd11cf-pnqtg"
Sep 16 12:56:37.955283 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:37.955255    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19dd11cf-pnqtg"]
Sep 16 12:56:37.956863 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:56:37.956837    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08d797af_03c0_475a_b2ca_4838df8ac6b9.slice/crio-ec9eb4017314e13701c0b4d52b111e155396bb1a3c00753a4b778f0dcb212815 WatchSource:0}: Error finding container ec9eb4017314e13701c0b4d52b111e155396bb1a3c00753a4b778f0dcb212815: Status 404 returned error can't find the container with id ec9eb4017314e13701c0b4d52b111e155396bb1a3c00753a4b778f0dcb212815
Sep 16 12:56:38.348936 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:38.348898    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19dd11cf-pnqtg" event={"ID":"08d797af-03c0-475a-b2ca-4838df8ac6b9","Type":"ContainerStarted","Data":"4557f9176cb29640260a579f1448633d35227028bc72b389174147c2dbf26af4"}
Sep 16 12:56:38.349118 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:38.348951    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19dd11cf-pnqtg" event={"ID":"08d797af-03c0-475a-b2ca-4838df8ac6b9","Type":"ContainerStarted","Data":"ec9eb4017314e13701c0b4d52b111e155396bb1a3c00753a4b778f0dcb212815"}
Sep 16 12:56:38.365021 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:38.364959    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19dd11cf-pnqtg" podStartSLOduration=1.364940475 podStartE2EDuration="1.364940475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:56:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:56:38.364562576 +0000 UTC m=+1104.811795217" watchObservedRunningTime="2026-09-16 12:56:38.364940475 +0000 UTC m=+1104.812173116"
Sep 16 12:56:38.602428 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:38.602387    2577 patch_prober.go:28] interesting pod/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 16 12:56:38.602633 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:38.602467    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9" podUID="cba4736f-ac79-494d-95db-4cf94129d658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 16 12:56:39.356125 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:39.356089    2577 generic.go:356] "Generic (PLEG): container finished" podID="08d797af-03c0-475a-b2ca-4838df8ac6b9" containerID="4557f9176cb29640260a579f1448633d35227028bc72b389174147c2dbf26af4" exitCode=0
Sep 16 12:56:39.356582 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:39.356209    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19dd11cf-pnqtg" event={"ID":"08d797af-03c0-475a-b2ca-4838df8ac6b9","Type":"ContainerDied","Data":"4557f9176cb29640260a579f1448633d35227028bc72b389174147c2dbf26af4"}
Sep 16 12:56:39.602527 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:39.602469    2577 patch_prober.go:28] interesting pod/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 16 12:56:39.602790 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:39.602757    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9" podUID="cba4736f-ac79-494d-95db-4cf94129d658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 16 12:56:40.516142 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:40.516103    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19dd11cf-pnqtg"
Sep 16 12:56:40.580284 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:40.580254    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08d797af-03c0-475a-b2ca-4838df8ac6b9-maas-api-service-ca\") pod \"08d797af-03c0-475a-b2ca-4838df8ac6b9\" (UID: \"08d797af-03c0-475a-b2ca-4838df8ac6b9\") "
Sep 16 12:56:40.580623 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:40.580597    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08d797af-03c0-475a-b2ca-4838df8ac6b9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "08d797af-03c0-475a-b2ca-4838df8ac6b9" (UID: "08d797af-03c0-475a-b2ca-4838df8ac6b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:56:40.606271 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:40.606244    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"
Sep 16 12:56:40.606675 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:40.606657    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"
Sep 16 12:56:40.607388 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:40.607369    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"
Sep 16 12:56:40.682217 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:40.682131    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08d797af-03c0-475a-b2ca-4838df8ac6b9-maas-api-service-ca\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:56:41.369694 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:41.369664    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19dd11cf-pnqtg"
Sep 16 12:56:41.369861 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:41.369696    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19dd11cf-pnqtg" event={"ID":"08d797af-03c0-475a-b2ca-4838df8ac6b9","Type":"ContainerDied","Data":"ec9eb4017314e13701c0b4d52b111e155396bb1a3c00753a4b778f0dcb212815"}
Sep 16 12:56:41.369861 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:41.369742    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec9eb4017314e13701c0b4d52b111e155396bb1a3c00753a4b778f0dcb212815"
Sep 16 12:56:43.897751 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:43.897714    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-19dd11cf-84b6456898-zfxm2"]
Sep 16 12:56:43.898289 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:43.897975    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-19dd11cf-84b6456898-zfxm2" podUID="1072d0fd-8c68-414f-a74a-44ac8414d6c1" containerName="payload-pre-processing" containerID="cri-o://5a631c1accd84be189372a503dbfbe986e1297534a2d2055933b67077527440a" gracePeriod=30
Sep 16 12:56:44.111683 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:44.111640    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19dd11cf-pnqtg"]
Sep 16 12:56:44.115251 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:44.115215    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19dd11cf-pnqtg"]
Sep 16 12:56:44.189329 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:44.189287    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-19dd11cf-84b6456898-zfxm2"
Sep 16 12:56:44.223284 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:44.223251    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9ph2\" (UniqueName: \"kubernetes.io/projected/1072d0fd-8c68-414f-a74a-44ac8414d6c1-kube-api-access-n9ph2\") pod \"1072d0fd-8c68-414f-a74a-44ac8414d6c1\" (UID: \"1072d0fd-8c68-414f-a74a-44ac8414d6c1\") "
Sep 16 12:56:44.223418 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:44.223338    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1072d0fd-8c68-414f-a74a-44ac8414d6c1-plugins-config-volume\") pod \"1072d0fd-8c68-414f-a74a-44ac8414d6c1\" (UID: \"1072d0fd-8c68-414f-a74a-44ac8414d6c1\") "
Sep 16 12:56:44.223684 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:44.223657    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1072d0fd-8c68-414f-a74a-44ac8414d6c1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1072d0fd-8c68-414f-a74a-44ac8414d6c1" (UID: "1072d0fd-8c68-414f-a74a-44ac8414d6c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:56:44.223805 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:44.223791    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1072d0fd-8c68-414f-a74a-44ac8414d6c1-plugins-config-volume\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:56:44.225532 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:44.225504    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1072d0fd-8c68-414f-a74a-44ac8414d6c1-kube-api-access-n9ph2" (OuterVolumeSpecName: "kube-api-access-n9ph2") pod "1072d0fd-8c68-414f-a74a-44ac8414d6c1" (UID: "1072d0fd-8c68-414f-a74a-44ac8414d6c1"). InnerVolumeSpecName "kube-api-access-n9ph2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:56:44.325156 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:44.325119    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9ph2\" (UniqueName: \"kubernetes.io/projected/1072d0fd-8c68-414f-a74a-44ac8414d6c1-kube-api-access-n9ph2\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:56:44.386885 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:44.386852    2577 generic.go:356] "Generic (PLEG): container finished" podID="1072d0fd-8c68-414f-a74a-44ac8414d6c1" containerID="5a631c1accd84be189372a503dbfbe986e1297534a2d2055933b67077527440a" exitCode=0
Sep 16 12:56:44.387119 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:44.386900    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-19dd11cf-84b6456898-zfxm2"
Sep 16 12:56:44.387119 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:44.386903    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-19dd11cf-84b6456898-zfxm2" event={"ID":"1072d0fd-8c68-414f-a74a-44ac8414d6c1","Type":"ContainerDied","Data":"5a631c1accd84be189372a503dbfbe986e1297534a2d2055933b67077527440a"}
Sep 16 12:56:44.387119 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:44.386971    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-19dd11cf-84b6456898-zfxm2" event={"ID":"1072d0fd-8c68-414f-a74a-44ac8414d6c1","Type":"ContainerDied","Data":"0e67626b12a6213b9bea02729a3faa3a4b4bbee201268db6316736cc80a9f5d5"}
Sep 16 12:56:44.387119 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:44.387000    2577 scope.go:117] "RemoveContainer" containerID="5a631c1accd84be189372a503dbfbe986e1297534a2d2055933b67077527440a"
Sep 16 12:56:44.396841 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:44.396824    2577 scope.go:117] "RemoveContainer" containerID="5a631c1accd84be189372a503dbfbe986e1297534a2d2055933b67077527440a"
Sep 16 12:56:44.397167 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:56:44.397148    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a631c1accd84be189372a503dbfbe986e1297534a2d2055933b67077527440a\": container with ID starting with 5a631c1accd84be189372a503dbfbe986e1297534a2d2055933b67077527440a not found: ID does not exist" containerID="5a631c1accd84be189372a503dbfbe986e1297534a2d2055933b67077527440a"
Sep 16 12:56:44.397235 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:44.397174    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a631c1accd84be189372a503dbfbe986e1297534a2d2055933b67077527440a"} err="failed to get container status \"5a631c1accd84be189372a503dbfbe986e1297534a2d2055933b67077527440a\": rpc error: code = NotFound desc = could not find container \"5a631c1accd84be189372a503dbfbe986e1297534a2d2055933b67077527440a\": container with ID starting with 5a631c1accd84be189372a503dbfbe986e1297534a2d2055933b67077527440a not found: ID does not exist"
Sep 16 12:56:44.411847 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:44.411825    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-19dd11cf-84b6456898-zfxm2"]
Sep 16 12:56:44.416943 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:44.416897    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-19dd11cf-84b6456898-zfxm2"]
Sep 16 12:56:45.224044 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:45.223996    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"]
Sep 16 12:56:45.224474 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:45.224296    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9" podUID="cba4736f-ac79-494d-95db-4cf94129d658" containerName="istio-proxy" containerID="cri-o://b5e38ae32feea2958122f1b68123bc268400482fb27cdde219538c1ffabd0f81" gracePeriod=30
Sep 16 12:56:45.965550 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:45.965515    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"]
Sep 16 12:56:45.966090 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:45.966068    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08d797af-03c0-475a-b2ca-4838df8ac6b9" containerName="revoke-keys"
Sep 16 12:56:45.966090 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:45.966091    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="08d797af-03c0-475a-b2ca-4838df8ac6b9" containerName="revoke-keys"
Sep 16 12:56:45.966288 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:45.966111    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1072d0fd-8c68-414f-a74a-44ac8414d6c1" containerName="payload-pre-processing"
Sep 16 12:56:45.966288 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:45.966119    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1072d0fd-8c68-414f-a74a-44ac8414d6c1" containerName="payload-pre-processing"
Sep 16 12:56:45.966288 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:45.966230    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1072d0fd-8c68-414f-a74a-44ac8414d6c1" containerName="payload-pre-processing"
Sep 16 12:56:45.966288 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:45.966243    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="08d797af-03c0-475a-b2ca-4838df8ac6b9" containerName="revoke-keys"
Sep 16 12:56:45.970846 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:45.970823    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"
Sep 16 12:56:45.973761 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:45.973741    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-d0f5a035-openshift-default-dockercfg-bz77s\""
Sep 16 12:56:45.981185 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:45.981162    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"]
Sep 16 12:56:46.044257 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:46.044224    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6a45ec-0e42-470e-800f-b543a32fdd3b-istio-podinfo\") pod \"e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf\" (UID: \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\") " pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"
Sep 16 12:56:46.044429 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:46.044264    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx2ns\" (UniqueName: \"kubernetes.io/projected/7c6a45ec-0e42-470e-800f-b543a32fdd3b-kube-api-access-tx2ns\") pod \"e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf\" (UID: \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\") " pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"
Sep 16 12:56:46.044429 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:46.044297    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6a45ec-0e42-470e-800f-b543a32fdd3b-workload-certs\") pod \"e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf\" (UID: \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\") " pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"
Sep 16 12:56:46.044429 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:46.044410    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6a45ec-0e42-470e-800f-b543a32fdd3b-istio-envoy\") pod \"e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf\" (UID: \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\") " pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"
Sep 16 12:56:46.044607 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:46.044454    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6a45ec-0e42-470e-800f-b543a32fdd3b-workload-socket\") pod \"e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf\" (UID: \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\") " pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"
Sep 16 12:56:46.044607 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:46.044535    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6a45ec-0e42-470e-800f-b543a32fdd3b-istiod-ca-cert\") pod \"e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf\" (UID: \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\") " pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"
Sep 16 12:56:46.044607 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:46.044593    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6a45ec-0e42-470e-800f-b543a32fdd3b-credential-socket\") pod \"e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf\" (UID: \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\") " pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"
Sep 16 12:56:46.044754 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:46.044632    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6a45ec-0e42-470e-800f-b543a32fdd3b-istio-data\") pod \"e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf\" (UID: \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\") " pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"
Sep 16 12:56:46.044754 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:46.044663    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c6a45ec-0e42-470e-800f-b543a32fdd3b-istio-token\") pod \"e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf\" (UID: \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\") " pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"
Sep 16 12:56:46.087060 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:46.086214    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08d797af-03c0-475a-b2ca-4838df8ac6b9" path="/var/lib/kubelet/pods/08d797af-03c0-475a-b2ca-4838df8ac6b9/volumes"
Sep 16 12:56:46.087060 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:46.086804    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1072d0fd-8c68-414f-a74a-44ac8414d6c1" path="/var/lib/kubelet/pods/1072d0fd-8c68-414f-a74a-44ac8414d6c1/volumes"
Sep 16 12:56:46.145494 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:46.145463    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6a45ec-0e42-470e-800f-b543a32fdd3b-istio-envoy\") pod \"e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf\" (UID: \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\") " pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"
Sep 16 12:56:46.145705 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:46.145509    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6a45ec-0e42-470e-800f-b543a32fdd3b-workload-socket\") pod \"e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf\" (UID: \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\") " pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"
Sep 16 12:56:46.145705 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:46.145556    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6a45ec-0e42-470e-800f-b543a32fdd3b-istiod-ca-cert\") pod \"e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf\" (UID: \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\") " pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"
Sep 16 12:56:46.145705 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:46.145641    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6a45ec-0e42-470e-800f-b543a32fdd3b-credential-socket\") pod \"e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf\" (UID: \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\") " pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"
Sep 16 12:56:46.145705 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:46.145677    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6a45ec-0e42-470e-800f-b543a32fdd3b-istio-data\") pod \"e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf\" (UID: \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\") " pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"
Sep 16 12:56:46.145965 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:46.145710    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c6a45ec-0e42-470e-800f-b543a32fdd3b-istio-token\") pod \"e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf\" (UID: \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\") " pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"
Sep 16 12:56:46.145965 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:46.145758    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6a45ec-0e42-470e-800f-b543a32fdd3b-istio-podinfo\") pod \"e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf\" (UID: \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\") " pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"
Sep 16 12:56:46.145965 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:46.145817    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx2ns\" (UniqueName: \"kubernetes.io/projected/7c6a45ec-0e42-470e-800f-b543a32fdd3b-kube-api-access-tx2ns\") pod \"e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf\" (UID: \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\") " pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"
Sep 16 12:56:46.145965 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:46.145883    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6a45ec-0e42-470e-800f-b543a32fdd3b-workload-certs\") pod \"e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf\" (UID: \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\") " pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"
Sep 16 12:56:46.145965 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:46.145949    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6a45ec-0e42-470e-800f-b543a32fdd3b-workload-socket\") pod \"e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf\" (UID: \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\") " pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"
Sep 16 12:56:46.146239 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:46.146007    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6a45ec-0e42-470e-800f-b543a32fdd3b-credential-socket\") pod \"e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf\" (UID: \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\") " pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"
Sep 16 12:56:46.146239 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:46.146230    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6a45ec-0e42-470e-800f-b543a32fdd3b-workload-certs\") pod \"e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf\" (UID: \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\") " pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"
Sep 16 12:56:46.146344 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:46.146266    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6a45ec-0e42-470e-800f-b543a32fdd3b-istio-data\") pod \"e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf\" (UID: \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\") " pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"
Sep 16 12:56:46.146421 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:46.146402    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6a45ec-0e42-470e-800f-b543a32fdd3b-istiod-ca-cert\") pod \"e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf\" (UID: \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\") " pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"
Sep 16 12:56:46.148369 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:46.148339    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6a45ec-0e42-470e-800f-b543a32fdd3b-istio-envoy\") pod \"e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf\" (UID: \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\") " pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"
Sep 16 12:56:46.148478 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:46.148449    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6a45ec-0e42-470e-800f-b543a32fdd3b-istio-podinfo\") pod \"e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf\" (UID: \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\") " pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"
Sep 16 12:56:46.161232 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:46.161211    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c6a45ec-0e42-470e-800f-b543a32fdd3b-istio-token\") pod \"e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf\" (UID: \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\") " pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"
Sep 16 12:56:46.161406 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:46.161388    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx2ns\" (UniqueName: \"kubernetes.io/projected/7c6a45ec-0e42-470e-800f-b543a32fdd3b-kube-api-access-tx2ns\") pod \"e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf\" (UID: \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\") " pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"
Sep 16 12:56:46.282924 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:46.282894    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"
Sep 16 12:56:46.563239 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:46.563211    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"]
Sep 16 12:56:46.565181 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:56:46.565146    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c6a45ec_0e42_470e_800f_b543a32fdd3b.slice/crio-b8f4b28a7b6bc33fffbfe594e46f6fa350ae861837fb46d0159e07189bf4bd12 WatchSource:0}: Error finding container b8f4b28a7b6bc33fffbfe594e46f6fa350ae861837fb46d0159e07189bf4bd12: Status 404 returned error can't find the container with id b8f4b28a7b6bc33fffbfe594e46f6fa350ae861837fb46d0159e07189bf4bd12
Sep 16 12:56:46.567540 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:46.567510    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:56:46.567627 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:46.567576    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:56:46.567627 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:46.567602    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:56:47.410400 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:47.410370    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf" event={"ID":"7c6a45ec-0e42-470e-800f-b543a32fdd3b","Type":"ContainerStarted","Data":"8a8114590eed911cfe7bf100bd6522aa3c3779d9e284e600df3546acbee415a9"}
Sep 16 12:56:47.410804 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:47.410420    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf" event={"ID":"7c6a45ec-0e42-470e-800f-b543a32fdd3b","Type":"ContainerStarted","Data":"b8f4b28a7b6bc33fffbfe594e46f6fa350ae861837fb46d0159e07189bf4bd12"}
Sep 16 12:56:47.433597 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:47.433552    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf" podStartSLOduration=2.433538205 podStartE2EDuration="2.433538205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:56:47.430916526 +0000 UTC m=+1113.878149168" watchObservedRunningTime="2026-09-16 12:56:47.433538205 +0000 UTC m=+1113.880770846"
Sep 16 12:56:48.283483 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:48.283450    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"
Sep 16 12:56:48.288848 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:48.288820    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"
Sep 16 12:56:48.416822 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:48.416787    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"
Sep 16 12:56:48.417838 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:48.417816    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"
Sep 16 12:56:50.430480 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:50.430445    2577 generic.go:356] "Generic (PLEG): container finished" podID="cba4736f-ac79-494d-95db-4cf94129d658" containerID="b5e38ae32feea2958122f1b68123bc268400482fb27cdde219538c1ffabd0f81" exitCode=0
Sep 16 12:56:50.430480 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:50.430476    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9" event={"ID":"cba4736f-ac79-494d-95db-4cf94129d658","Type":"ContainerDied","Data":"b5e38ae32feea2958122f1b68123bc268400482fb27cdde219538c1ffabd0f81"}
Sep 16 12:56:50.430897 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:50.430539    2577 scope.go:117] "RemoveContainer" containerID="808c9466723b1d04a9570a3663996c45da9a26df0efe1db595fa6cca010940e4"
Sep 16 12:56:50.480070 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:50.480050    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"
Sep 16 12:56:50.593327 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:50.593289    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frcvv\" (UniqueName: \"kubernetes.io/projected/cba4736f-ac79-494d-95db-4cf94129d658-kube-api-access-frcvv\") pod \"cba4736f-ac79-494d-95db-4cf94129d658\" (UID: \"cba4736f-ac79-494d-95db-4cf94129d658\") "
Sep 16 12:56:50.593327 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:50.593332    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cba4736f-ac79-494d-95db-4cf94129d658-istio-token\") pod \"cba4736f-ac79-494d-95db-4cf94129d658\" (UID: \"cba4736f-ac79-494d-95db-4cf94129d658\") "
Sep 16 12:56:50.593656 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:50.593357    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cba4736f-ac79-494d-95db-4cf94129d658-istiod-ca-cert\") pod \"cba4736f-ac79-494d-95db-4cf94129d658\" (UID: \"cba4736f-ac79-494d-95db-4cf94129d658\") "
Sep 16 12:56:50.593656 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:50.593434    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba4736f-ac79-494d-95db-4cf94129d658-credential-socket\") pod \"cba4736f-ac79-494d-95db-4cf94129d658\" (UID: \"cba4736f-ac79-494d-95db-4cf94129d658\") "
Sep 16 12:56:50.593656 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:50.593462    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cba4736f-ac79-494d-95db-4cf94129d658-workload-certs\") pod \"cba4736f-ac79-494d-95db-4cf94129d658\" (UID: \"cba4736f-ac79-494d-95db-4cf94129d658\") "
Sep 16 12:56:50.593656 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:50.593519    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cba4736f-ac79-494d-95db-4cf94129d658-istio-data\") pod \"cba4736f-ac79-494d-95db-4cf94129d658\" (UID: \"cba4736f-ac79-494d-95db-4cf94129d658\") "
Sep 16 12:56:50.593656 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:50.593544    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cba4736f-ac79-494d-95db-4cf94129d658-istio-envoy\") pod \"cba4736f-ac79-494d-95db-4cf94129d658\" (UID: \"cba4736f-ac79-494d-95db-4cf94129d658\") "
Sep 16 12:56:50.593656 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:50.593628    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cba4736f-ac79-494d-95db-4cf94129d658-istio-podinfo\") pod \"cba4736f-ac79-494d-95db-4cf94129d658\" (UID: \"cba4736f-ac79-494d-95db-4cf94129d658\") "
Sep 16 12:56:50.593962 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:50.593661    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba4736f-ac79-494d-95db-4cf94129d658-workload-socket\") pod \"cba4736f-ac79-494d-95db-4cf94129d658\" (UID: \"cba4736f-ac79-494d-95db-4cf94129d658\") "
Sep 16 12:56:50.594597 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:50.594154    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba4736f-ac79-494d-95db-4cf94129d658-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cba4736f-ac79-494d-95db-4cf94129d658" (UID: "cba4736f-ac79-494d-95db-4cf94129d658"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:56:50.594597 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:50.594310    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba4736f-ac79-494d-95db-4cf94129d658-istio-data" (OuterVolumeSpecName: "istio-data") pod "cba4736f-ac79-494d-95db-4cf94129d658" (UID: "cba4736f-ac79-494d-95db-4cf94129d658"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:56:50.594597 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:50.594474    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba4736f-ac79-494d-95db-4cf94129d658-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cba4736f-ac79-494d-95db-4cf94129d658" (UID: "cba4736f-ac79-494d-95db-4cf94129d658"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:56:50.594597 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:50.594487    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba4736f-ac79-494d-95db-4cf94129d658-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cba4736f-ac79-494d-95db-4cf94129d658" (UID: "cba4736f-ac79-494d-95db-4cf94129d658"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:56:50.594597 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:50.594507    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cba4736f-ac79-494d-95db-4cf94129d658-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cba4736f-ac79-494d-95db-4cf94129d658" (UID: "cba4736f-ac79-494d-95db-4cf94129d658"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:56:50.595889 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:50.595851    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cba4736f-ac79-494d-95db-4cf94129d658-istio-token" (OuterVolumeSpecName: "istio-token") pod "cba4736f-ac79-494d-95db-4cf94129d658" (UID: "cba4736f-ac79-494d-95db-4cf94129d658"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:56:50.596177 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:50.596152    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cba4736f-ac79-494d-95db-4cf94129d658-kube-api-access-frcvv" (OuterVolumeSpecName: "kube-api-access-frcvv") pod "cba4736f-ac79-494d-95db-4cf94129d658" (UID: "cba4736f-ac79-494d-95db-4cf94129d658"). InnerVolumeSpecName "kube-api-access-frcvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:56:50.596325 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:50.596299    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba4736f-ac79-494d-95db-4cf94129d658-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cba4736f-ac79-494d-95db-4cf94129d658" (UID: "cba4736f-ac79-494d-95db-4cf94129d658"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:56:50.596620 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:50.596600    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cba4736f-ac79-494d-95db-4cf94129d658-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cba4736f-ac79-494d-95db-4cf94129d658" (UID: "cba4736f-ac79-494d-95db-4cf94129d658"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:56:50.695614 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:50.695569    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba4736f-ac79-494d-95db-4cf94129d658-workload-socket\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:56:50.695614 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:50.695599    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frcvv\" (UniqueName: \"kubernetes.io/projected/cba4736f-ac79-494d-95db-4cf94129d658-kube-api-access-frcvv\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:56:50.695614 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:50.695609    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cba4736f-ac79-494d-95db-4cf94129d658-istio-token\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:56:50.695614 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:50.695618    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cba4736f-ac79-494d-95db-4cf94129d658-istiod-ca-cert\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:56:50.695614 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:50.695626    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba4736f-ac79-494d-95db-4cf94129d658-credential-socket\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:56:50.696074 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:50.695635    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cba4736f-ac79-494d-95db-4cf94129d658-workload-certs\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:56:50.696074 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:50.695642    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cba4736f-ac79-494d-95db-4cf94129d658-istio-data\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:56:50.696074 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:50.695650    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cba4736f-ac79-494d-95db-4cf94129d658-istio-envoy\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:56:50.696074 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:50.695657    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cba4736f-ac79-494d-95db-4cf94129d658-istio-podinfo\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:56:51.438874 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:51.438840    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"
Sep 16 12:56:51.439326 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:51.438878    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9" event={"ID":"cba4736f-ac79-494d-95db-4cf94129d658","Type":"ContainerDied","Data":"cf7c3fd938ede7214305fc232c978b92b6b1cdf4fb609f93923db70a1ae0dc50"}
Sep 16 12:56:51.439326 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:51.438937    2577 scope.go:117] "RemoveContainer" containerID="b5e38ae32feea2958122f1b68123bc268400482fb27cdde219538c1ffabd0f81"
Sep 16 12:56:51.469987 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:51.469955    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"]
Sep 16 12:56:51.473726 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:51.473706    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-19dd11cf-openshift-default-6bd7b4bb6b-f5ch9"]
Sep 16 12:56:52.055530 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:52.055497    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cba4736f-ac79-494d-95db-4cf94129d658" path="/var/lib/kubelet/pods/cba4736f-ac79-494d-95db-4cf94129d658/volumes"
Sep 16 12:56:54.356733 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:54.356698    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d0f5a035-54c4f7fd98-hxf8s"]
Sep 16 12:56:54.357242 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:54.357223    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cba4736f-ac79-494d-95db-4cf94129d658" containerName="istio-proxy"
Sep 16 12:56:54.357311 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:54.357244    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cba4736f-ac79-494d-95db-4cf94129d658" containerName="istio-proxy"
Sep 16 12:56:54.357311 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:54.357264    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cba4736f-ac79-494d-95db-4cf94129d658" containerName="istio-proxy"
Sep 16 12:56:54.357311 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:54.357269    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cba4736f-ac79-494d-95db-4cf94129d658" containerName="istio-proxy"
Sep 16 12:56:54.357418 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:54.357364    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cba4736f-ac79-494d-95db-4cf94129d658" containerName="istio-proxy"
Sep 16 12:56:54.361869 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:54.361851    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d0f5a035-54c4f7fd98-hxf8s"
Sep 16 12:56:54.364376 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:54.364350    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-d0f5a035-dockercfg-cbmq8\""
Sep 16 12:56:54.364496 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:54.364356    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-d0f5a035\""
Sep 16 12:56:54.372071 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:54.372049    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d0f5a035-54c4f7fd98-hxf8s"]
Sep 16 12:56:54.437969 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:54.437934    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc9hr\" (UniqueName: \"kubernetes.io/projected/46d533d7-d229-4daf-97de-f016c975177b-kube-api-access-wc9hr\") pod \"payload-pre-processing-e2e-derive-d0f5a035-54c4f7fd98-hxf8s\" (UID: \"46d533d7-d229-4daf-97de-f016c975177b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d0f5a035-54c4f7fd98-hxf8s"
Sep 16 12:56:54.438135 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:54.437997    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46d533d7-d229-4daf-97de-f016c975177b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-d0f5a035-54c4f7fd98-hxf8s\" (UID: \"46d533d7-d229-4daf-97de-f016c975177b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d0f5a035-54c4f7fd98-hxf8s"
Sep 16 12:56:54.539532 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:54.539487    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc9hr\" (UniqueName: \"kubernetes.io/projected/46d533d7-d229-4daf-97de-f016c975177b-kube-api-access-wc9hr\") pod \"payload-pre-processing-e2e-derive-d0f5a035-54c4f7fd98-hxf8s\" (UID: \"46d533d7-d229-4daf-97de-f016c975177b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d0f5a035-54c4f7fd98-hxf8s"
Sep 16 12:56:54.539738 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:54.539578    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46d533d7-d229-4daf-97de-f016c975177b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-d0f5a035-54c4f7fd98-hxf8s\" (UID: \"46d533d7-d229-4daf-97de-f016c975177b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d0f5a035-54c4f7fd98-hxf8s"
Sep 16 12:56:54.540435 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:54.540399    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46d533d7-d229-4daf-97de-f016c975177b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-d0f5a035-54c4f7fd98-hxf8s\" (UID: \"46d533d7-d229-4daf-97de-f016c975177b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d0f5a035-54c4f7fd98-hxf8s"
Sep 16 12:56:54.548073 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:54.548012    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc9hr\" (UniqueName: \"kubernetes.io/projected/46d533d7-d229-4daf-97de-f016c975177b-kube-api-access-wc9hr\") pod \"payload-pre-processing-e2e-derive-d0f5a035-54c4f7fd98-hxf8s\" (UID: \"46d533d7-d229-4daf-97de-f016c975177b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d0f5a035-54c4f7fd98-hxf8s"
Sep 16 12:56:54.673875 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:54.673785    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d0f5a035-54c4f7fd98-hxf8s"
Sep 16 12:56:54.950171 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:54.950144    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d0f5a035-54c4f7fd98-hxf8s"]
Sep 16 12:56:54.952061 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:56:54.952007    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46d533d7_d229_4daf_97de_f016c975177b.slice/crio-ca6cef112639e7bf00171abf897088d6245a75007d6edf482b850bfcf99465d5 WatchSource:0}: Error finding container ca6cef112639e7bf00171abf897088d6245a75007d6edf482b850bfcf99465d5: Status 404 returned error can't find the container with id ca6cef112639e7bf00171abf897088d6245a75007d6edf482b850bfcf99465d5
Sep 16 12:56:55.480151 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:55.480103    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d0f5a035-54c4f7fd98-hxf8s" event={"ID":"46d533d7-d229-4daf-97de-f016c975177b","Type":"ContainerStarted","Data":"9078b80a3b7d4e3e5b45ae0097971e0f56f52872bc51da3137eaeb416e9106f7"}
Sep 16 12:56:55.480536 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:55.480173    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d0f5a035-54c4f7fd98-hxf8s" event={"ID":"46d533d7-d229-4daf-97de-f016c975177b","Type":"ContainerStarted","Data":"ca6cef112639e7bf00171abf897088d6245a75007d6edf482b850bfcf99465d5"}
Sep 16 12:56:55.480536 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:55.480319    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-d0f5a035-54c4f7fd98-hxf8s"
Sep 16 12:56:55.514138 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:56:55.514084    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-d0f5a035-54c4f7fd98-hxf8s" podStartSLOduration=1.51406613 podStartE2EDuration="1.51406613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:56:55.511095967 +0000 UTC m=+1121.958328610" watchObservedRunningTime="2026-09-16 12:56:55.51406613 +0000 UTC m=+1121.961298774"
Sep 16 12:57:06.487969 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:06.487936    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-d0f5a035-54c4f7fd98-hxf8s"
Sep 16 12:57:11.585431 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:11.585344    2577 generic.go:356] "Generic (PLEG): container finished" podID="7c6a45ec-0e42-470e-800f-b543a32fdd3b" containerID="8a8114590eed911cfe7bf100bd6522aa3c3779d9e284e600df3546acbee415a9" exitCode=0
Sep 16 12:57:11.585431 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:11.585411    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf" event={"ID":"7c6a45ec-0e42-470e-800f-b543a32fdd3b","Type":"ContainerDied","Data":"8a8114590eed911cfe7bf100bd6522aa3c3779d9e284e600df3546acbee415a9"}
Sep 16 12:57:11.585989 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:11.585966    2577 scope.go:117] "RemoveContainer" containerID="8a8114590eed911cfe7bf100bd6522aa3c3779d9e284e600df3546acbee415a9"
Sep 16 12:57:11.587310 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:11.587279    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:57:11.587434 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:11.587345    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:57:11.587434 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:11.587376    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 12:57:12.597349 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:12.597299    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf" event={"ID":"7c6a45ec-0e42-470e-800f-b543a32fdd3b","Type":"ContainerStarted","Data":"f55ce6d5f67143f24ecf087773640c358432185cdf9317b5e5644f5607080e2d"}
Sep 16 12:57:13.283769 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:13.283740    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"
Sep 16 12:57:13.283981 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:13.283958    2577 patch_prober.go:28] interesting pod/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 16 12:57:13.284107 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:13.284068    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf" podUID="7c6a45ec-0e42-470e-800f-b543a32fdd3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 16 12:57:13.669932 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:13.669855    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-a24d8b/autores-model-a24d8b-kserve-5878448ddc-jgxwh"]
Sep 16 12:57:13.670422 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:13.670408    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cba4736f-ac79-494d-95db-4cf94129d658" containerName="istio-proxy"
Sep 16 12:57:13.673613 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:13.673598    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a24d8b/autores-model-a24d8b-kserve-5878448ddc-jgxwh"
Sep 16 12:57:13.677525 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:13.677460    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-a24d8b\"/\"kube-root-ca.crt\""
Sep 16 12:57:13.677525 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:13.677468    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-a24d8b\"/\"default-dockercfg-9p2wr\""
Sep 16 12:57:13.677525 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:13.677524    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-a24d8b\"/\"openshift-service-ca.crt\""
Sep 16 12:57:13.677817 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:13.677766    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-a24d8b\"/\"autores-model-a24d8b-kserve-self-signed-certs\""
Sep 16 12:57:13.683504 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:13.683482    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-a24d8b/autores-model-a24d8b-kserve-5878448ddc-jgxwh"]
Sep 16 12:57:13.747402 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:13.747358    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljs5j\" (UniqueName: \"kubernetes.io/projected/2a6e48b9-3e76-41c9-8c6e-b12c47c791c6-kube-api-access-ljs5j\") pod \"autores-model-a24d8b-kserve-5878448ddc-jgxwh\" (UID: \"2a6e48b9-3e76-41c9-8c6e-b12c47c791c6\") " pod="ai-tenant-e2e-autores-a24d8b/autores-model-a24d8b-kserve-5878448ddc-jgxwh"
Sep 16 12:57:13.747584 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:13.747410    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a6e48b9-3e76-41c9-8c6e-b12c47c791c6-tmp-dir\") pod \"autores-model-a24d8b-kserve-5878448ddc-jgxwh\" (UID: \"2a6e48b9-3e76-41c9-8c6e-b12c47c791c6\") " pod="ai-tenant-e2e-autores-a24d8b/autores-model-a24d8b-kserve-5878448ddc-jgxwh"
Sep 16 12:57:13.747584 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:13.747474    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a6e48b9-3e76-41c9-8c6e-b12c47c791c6-model-cache\") pod \"autores-model-a24d8b-kserve-5878448ddc-jgxwh\" (UID: \"2a6e48b9-3e76-41c9-8c6e-b12c47c791c6\") " pod="ai-tenant-e2e-autores-a24d8b/autores-model-a24d8b-kserve-5878448ddc-jgxwh"
Sep 16 12:57:13.747584 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:13.747517    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a6e48b9-3e76-41c9-8c6e-b12c47c791c6-dshm\") pod \"autores-model-a24d8b-kserve-5878448ddc-jgxwh\" (UID: \"2a6e48b9-3e76-41c9-8c6e-b12c47c791c6\") " pod="ai-tenant-e2e-autores-a24d8b/autores-model-a24d8b-kserve-5878448ddc-jgxwh"
Sep 16 12:57:13.747584 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:13.747567    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a6e48b9-3e76-41c9-8c6e-b12c47c791c6-tls-certs\") pod \"autores-model-a24d8b-kserve-5878448ddc-jgxwh\" (UID: \"2a6e48b9-3e76-41c9-8c6e-b12c47c791c6\") " pod="ai-tenant-e2e-autores-a24d8b/autores-model-a24d8b-kserve-5878448ddc-jgxwh"
Sep 16 12:57:13.747734 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:13.747621    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a6e48b9-3e76-41c9-8c6e-b12c47c791c6-home\") pod \"autores-model-a24d8b-kserve-5878448ddc-jgxwh\" (UID: \"2a6e48b9-3e76-41c9-8c6e-b12c47c791c6\") " pod="ai-tenant-e2e-autores-a24d8b/autores-model-a24d8b-kserve-5878448ddc-jgxwh"
Sep 16 12:57:13.849545 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:13.849195    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljs5j\" (UniqueName: \"kubernetes.io/projected/2a6e48b9-3e76-41c9-8c6e-b12c47c791c6-kube-api-access-ljs5j\") pod \"autores-model-a24d8b-kserve-5878448ddc-jgxwh\" (UID: \"2a6e48b9-3e76-41c9-8c6e-b12c47c791c6\") " pod="ai-tenant-e2e-autores-a24d8b/autores-model-a24d8b-kserve-5878448ddc-jgxwh"
Sep 16 12:57:13.849545 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:13.849261    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a6e48b9-3e76-41c9-8c6e-b12c47c791c6-tmp-dir\") pod \"autores-model-a24d8b-kserve-5878448ddc-jgxwh\" (UID: \"2a6e48b9-3e76-41c9-8c6e-b12c47c791c6\") " pod="ai-tenant-e2e-autores-a24d8b/autores-model-a24d8b-kserve-5878448ddc-jgxwh"
Sep 16 12:57:13.849545 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:13.849301    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a6e48b9-3e76-41c9-8c6e-b12c47c791c6-model-cache\") pod \"autores-model-a24d8b-kserve-5878448ddc-jgxwh\" (UID: \"2a6e48b9-3e76-41c9-8c6e-b12c47c791c6\") " pod="ai-tenant-e2e-autores-a24d8b/autores-model-a24d8b-kserve-5878448ddc-jgxwh"
Sep 16 12:57:13.849545 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:13.849338    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a6e48b9-3e76-41c9-8c6e-b12c47c791c6-dshm\") pod \"autores-model-a24d8b-kserve-5878448ddc-jgxwh\" (UID: \"2a6e48b9-3e76-41c9-8c6e-b12c47c791c6\") " pod="ai-tenant-e2e-autores-a24d8b/autores-model-a24d8b-kserve-5878448ddc-jgxwh"
Sep 16 12:57:13.849545 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:13.849405    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a6e48b9-3e76-41c9-8c6e-b12c47c791c6-tls-certs\") pod \"autores-model-a24d8b-kserve-5878448ddc-jgxwh\" (UID: \"2a6e48b9-3e76-41c9-8c6e-b12c47c791c6\") " pod="ai-tenant-e2e-autores-a24d8b/autores-model-a24d8b-kserve-5878448ddc-jgxwh"
Sep 16 12:57:13.849545 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:13.849440    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a6e48b9-3e76-41c9-8c6e-b12c47c791c6-home\") pod \"autores-model-a24d8b-kserve-5878448ddc-jgxwh\" (UID: \"2a6e48b9-3e76-41c9-8c6e-b12c47c791c6\") " pod="ai-tenant-e2e-autores-a24d8b/autores-model-a24d8b-kserve-5878448ddc-jgxwh"
Sep 16 12:57:13.850008 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:13.849913    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a6e48b9-3e76-41c9-8c6e-b12c47c791c6-home\") pod \"autores-model-a24d8b-kserve-5878448ddc-jgxwh\" (UID: \"2a6e48b9-3e76-41c9-8c6e-b12c47c791c6\") " pod="ai-tenant-e2e-autores-a24d8b/autores-model-a24d8b-kserve-5878448ddc-jgxwh"
Sep 16 12:57:13.850224 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:13.850204    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a6e48b9-3e76-41c9-8c6e-b12c47c791c6-tmp-dir\") pod \"autores-model-a24d8b-kserve-5878448ddc-jgxwh\" (UID: \"2a6e48b9-3e76-41c9-8c6e-b12c47c791c6\") " pod="ai-tenant-e2e-autores-a24d8b/autores-model-a24d8b-kserve-5878448ddc-jgxwh"
Sep 16 12:57:13.850349 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:13.850331    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a6e48b9-3e76-41c9-8c6e-b12c47c791c6-model-cache\") pod \"autores-model-a24d8b-kserve-5878448ddc-jgxwh\" (UID: \"2a6e48b9-3e76-41c9-8c6e-b12c47c791c6\") " pod="ai-tenant-e2e-autores-a24d8b/autores-model-a24d8b-kserve-5878448ddc-jgxwh"
Sep 16 12:57:13.852831 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:13.852807    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a6e48b9-3e76-41c9-8c6e-b12c47c791c6-dshm\") pod \"autores-model-a24d8b-kserve-5878448ddc-jgxwh\" (UID: \"2a6e48b9-3e76-41c9-8c6e-b12c47c791c6\") " pod="ai-tenant-e2e-autores-a24d8b/autores-model-a24d8b-kserve-5878448ddc-jgxwh"
Sep 16 12:57:13.853594 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:13.853552    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a6e48b9-3e76-41c9-8c6e-b12c47c791c6-tls-certs\") pod \"autores-model-a24d8b-kserve-5878448ddc-jgxwh\" (UID: \"2a6e48b9-3e76-41c9-8c6e-b12c47c791c6\") " pod="ai-tenant-e2e-autores-a24d8b/autores-model-a24d8b-kserve-5878448ddc-jgxwh"
Sep 16 12:57:13.860110 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:13.860089    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljs5j\" (UniqueName: \"kubernetes.io/projected/2a6e48b9-3e76-41c9-8c6e-b12c47c791c6-kube-api-access-ljs5j\") pod \"autores-model-a24d8b-kserve-5878448ddc-jgxwh\" (UID: \"2a6e48b9-3e76-41c9-8c6e-b12c47c791c6\") " pod="ai-tenant-e2e-autores-a24d8b/autores-model-a24d8b-kserve-5878448ddc-jgxwh"
Sep 16 12:57:13.987409 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:13.987313    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-a24d8b\"/\"default-dockercfg-9p2wr\""
Sep 16 12:57:14.001154 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:14.001118    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a24d8b/autores-model-a24d8b-kserve-5878448ddc-jgxwh"
Sep 16 12:57:14.283343 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:14.283310    2577 patch_prober.go:28] interesting pod/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 16 12:57:14.283502 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:14.283365    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf" podUID="7c6a45ec-0e42-470e-800f-b543a32fdd3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 16 12:57:14.295059 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:14.295008    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-a24d8b/autores-model-a24d8b-kserve-5878448ddc-jgxwh"]
Sep 16 12:57:14.299846 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:57:14.299823    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a6e48b9_3e76_41c9_8c6e_b12c47c791c6.slice/crio-fdb3258a53cabe04fb9b3026d72b0e3caf07f20179f218e18106705dbd57fc2c WatchSource:0}: Error finding container fdb3258a53cabe04fb9b3026d72b0e3caf07f20179f218e18106705dbd57fc2c: Status 404 returned error can't find the container with id fdb3258a53cabe04fb9b3026d72b0e3caf07f20179f218e18106705dbd57fc2c
Sep 16 12:57:14.629252 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:14.629155    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a24d8b/autores-model-a24d8b-kserve-5878448ddc-jgxwh" event={"ID":"2a6e48b9-3e76-41c9-8c6e-b12c47c791c6","Type":"ContainerStarted","Data":"adb3f94d295786c7aa1538e8270f8c9a5a830fa331affc7c7e63a83b3de33e9b"}
Sep 16 12:57:14.629252 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:14.629217    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a24d8b/autores-model-a24d8b-kserve-5878448ddc-jgxwh" event={"ID":"2a6e48b9-3e76-41c9-8c6e-b12c47c791c6","Type":"ContainerStarted","Data":"fdb3258a53cabe04fb9b3026d72b0e3caf07f20179f218e18106705dbd57fc2c"}
Sep 16 12:57:14.645710 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:14.645658    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-a24d8b/autores-model-a24d8b-kserve-5878448ddc-jgxwh" podStartSLOduration=1.645639421 podStartE2EDuration="1.645639421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:57:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:57:14.645134782 +0000 UTC m=+1141.092367419" watchObservedRunningTime="2026-09-16 12:57:14.645639421 +0000 UTC m=+1141.092872065"
Sep 16 12:57:15.284100 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:15.284054    2577 patch_prober.go:28] interesting pod/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 16 12:57:15.284590 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:15.284123    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf" podUID="7c6a45ec-0e42-470e-800f-b543a32fdd3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 16 12:57:16.283675 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:16.283644    2577 patch_prober.go:28] interesting pod/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 16 12:57:16.283853 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:16.283699    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf" podUID="7c6a45ec-0e42-470e-800f-b543a32fdd3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 16 12:57:17.284249 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:17.284203    2577 patch_prober.go:28] interesting pod/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 16 12:57:17.284614 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:17.284273    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf" podUID="7c6a45ec-0e42-470e-800f-b543a32fdd3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 16 12:57:18.286841 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:18.286813    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"
Sep 16 12:57:18.287286 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:18.287082    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"
Sep 16 12:57:18.288015 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:18.287995    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"
Sep 16 12:57:18.770433 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:18.770400    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d0f5a035-7n4q5"]
Sep 16 12:57:18.775415 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:18.775399    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d0f5a035-7n4q5"
Sep 16 12:57:18.778120 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:18.778098    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-64zph\""
Sep 16 12:57:18.778235 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:18.778121    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:57:18.782701 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:18.782676    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d0f5a035-7n4q5"]
Sep 16 12:57:18.915221 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:18.915184    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70b85bf6-b85a-4f41-9cba-29f2ae53377d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d0f5a035-7n4q5\" (UID: \"70b85bf6-b85a-4f41-9cba-29f2ae53377d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d0f5a035-7n4q5"
Sep 16 12:57:19.015923 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:19.015891    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70b85bf6-b85a-4f41-9cba-29f2ae53377d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d0f5a035-7n4q5\" (UID: \"70b85bf6-b85a-4f41-9cba-29f2ae53377d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d0f5a035-7n4q5"
Sep 16 12:57:19.016517 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:19.016498    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70b85bf6-b85a-4f41-9cba-29f2ae53377d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d0f5a035-7n4q5\" (UID: \"70b85bf6-b85a-4f41-9cba-29f2ae53377d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d0f5a035-7n4q5"
Sep 16 12:57:19.090518 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:19.090444    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d0f5a035-7n4q5"
Sep 16 12:57:19.362364 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:19.362341    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d0f5a035-7n4q5"]
Sep 16 12:57:19.364665 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:57:19.364635    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b85bf6_b85a_4f41_9cba_29f2ae53377d.slice/crio-27c6f48a892d4a2a7fe00fc329a587d0935f71a5edc2e901e689c9ba5c13de2d WatchSource:0}: Error finding container 27c6f48a892d4a2a7fe00fc329a587d0935f71a5edc2e901e689c9ba5c13de2d: Status 404 returned error can't find the container with id 27c6f48a892d4a2a7fe00fc329a587d0935f71a5edc2e901e689c9ba5c13de2d
Sep 16 12:57:19.664020 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:19.663979    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d0f5a035-7n4q5" event={"ID":"70b85bf6-b85a-4f41-9cba-29f2ae53377d","Type":"ContainerStarted","Data":"fb2cb5a28df9d9099392e5f855128f1c166941bb9e3d56911a0f3ab361bffd06"}
Sep 16 12:57:19.664208 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:19.664074    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d0f5a035-7n4q5" event={"ID":"70b85bf6-b85a-4f41-9cba-29f2ae53377d","Type":"ContainerStarted","Data":"27c6f48a892d4a2a7fe00fc329a587d0935f71a5edc2e901e689c9ba5c13de2d"}
Sep 16 12:57:19.682551 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:19.682508    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d0f5a035-7n4q5" podStartSLOduration=1.6824923 podStartE2EDuration="1.6824923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:57:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:57:19.680165433 +0000 UTC m=+1146.127398090" watchObservedRunningTime="2026-09-16 12:57:19.6824923 +0000 UTC m=+1146.129724941"
Sep 16 12:57:20.671812 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:20.671775    2577 generic.go:356] "Generic (PLEG): container finished" podID="70b85bf6-b85a-4f41-9cba-29f2ae53377d" containerID="fb2cb5a28df9d9099392e5f855128f1c166941bb9e3d56911a0f3ab361bffd06" exitCode=0
Sep 16 12:57:20.672206 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:20.671847    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d0f5a035-7n4q5" event={"ID":"70b85bf6-b85a-4f41-9cba-29f2ae53377d","Type":"ContainerDied","Data":"fb2cb5a28df9d9099392e5f855128f1c166941bb9e3d56911a0f3ab361bffd06"}
Sep 16 12:57:21.823577 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:21.823552    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d0f5a035-7n4q5"
Sep 16 12:57:21.945374 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:21.945276    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70b85bf6-b85a-4f41-9cba-29f2ae53377d-maas-api-service-ca\") pod \"70b85bf6-b85a-4f41-9cba-29f2ae53377d\" (UID: \"70b85bf6-b85a-4f41-9cba-29f2ae53377d\") "
Sep 16 12:57:21.945618 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:21.945595    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70b85bf6-b85a-4f41-9cba-29f2ae53377d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "70b85bf6-b85a-4f41-9cba-29f2ae53377d" (UID: "70b85bf6-b85a-4f41-9cba-29f2ae53377d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:57:22.046620 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:22.046583    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70b85bf6-b85a-4f41-9cba-29f2ae53377d-maas-api-service-ca\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:57:22.684412 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:22.684384    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d0f5a035-7n4q5"
Sep 16 12:57:22.684593 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:22.684433    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d0f5a035-7n4q5" event={"ID":"70b85bf6-b85a-4f41-9cba-29f2ae53377d","Type":"ContainerDied","Data":"27c6f48a892d4a2a7fe00fc329a587d0935f71a5edc2e901e689c9ba5c13de2d"}
Sep 16 12:57:22.684593 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:22.684525    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27c6f48a892d4a2a7fe00fc329a587d0935f71a5edc2e901e689c9ba5c13de2d"
Sep 16 12:57:24.002274 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:24.002240    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-a24d8b/autores-model-a24d8b-kserve-5878448ddc-jgxwh"
Sep 16 12:57:24.002642 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:24.002379    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-a24d8b/autores-model-a24d8b-kserve-5878448ddc-jgxwh"
Sep 16 12:57:24.015068 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:24.015044    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-a24d8b/autores-model-a24d8b-kserve-5878448ddc-jgxwh"
Sep 16 12:57:24.710109 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:24.710082    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-a24d8b/autores-model-a24d8b-kserve-5878448ddc-jgxwh"
Sep 16 12:57:30.970102 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:30.970069    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d0f5a035-54c4f7fd98-hxf8s"]
Sep 16 12:57:30.970576 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:30.970291    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-d0f5a035-54c4f7fd98-hxf8s" podUID="46d533d7-d229-4daf-97de-f016c975177b" containerName="payload-pre-processing" containerID="cri-o://9078b80a3b7d4e3e5b45ae0097971e0f56f52872bc51da3137eaeb416e9106f7" gracePeriod=30
Sep 16 12:57:31.223833 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:31.223769    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d0f5a035-54c4f7fd98-hxf8s"
Sep 16 12:57:31.245939 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:31.245902    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46d533d7-d229-4daf-97de-f016c975177b-plugins-config-volume\") pod \"46d533d7-d229-4daf-97de-f016c975177b\" (UID: \"46d533d7-d229-4daf-97de-f016c975177b\") "
Sep 16 12:57:31.246107 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:31.245998    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wc9hr\" (UniqueName: \"kubernetes.io/projected/46d533d7-d229-4daf-97de-f016c975177b-kube-api-access-wc9hr\") pod \"46d533d7-d229-4daf-97de-f016c975177b\" (UID: \"46d533d7-d229-4daf-97de-f016c975177b\") "
Sep 16 12:57:31.246327 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:31.246301    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46d533d7-d229-4daf-97de-f016c975177b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "46d533d7-d229-4daf-97de-f016c975177b" (UID: "46d533d7-d229-4daf-97de-f016c975177b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:57:31.246419 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:31.246399    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46d533d7-d229-4daf-97de-f016c975177b-plugins-config-volume\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:57:31.248160 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:31.248135    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d533d7-d229-4daf-97de-f016c975177b-kube-api-access-wc9hr" (OuterVolumeSpecName: "kube-api-access-wc9hr") pod "46d533d7-d229-4daf-97de-f016c975177b" (UID: "46d533d7-d229-4daf-97de-f016c975177b"). InnerVolumeSpecName "kube-api-access-wc9hr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:57:31.294363 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:31.294317    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d0f5a035-7n4q5"]
Sep 16 12:57:31.296795 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:31.296768    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d0f5a035-7n4q5"]
Sep 16 12:57:31.347017 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:31.346983    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc9hr\" (UniqueName: \"kubernetes.io/projected/46d533d7-d229-4daf-97de-f016c975177b-kube-api-access-wc9hr\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:57:31.440636 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:31.440606    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"]
Sep 16 12:57:31.440848 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:31.440826    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf" podUID="7c6a45ec-0e42-470e-800f-b543a32fdd3b" containerName="istio-proxy" containerID="cri-o://f55ce6d5f67143f24ecf087773640c358432185cdf9317b5e5644f5607080e2d" gracePeriod=30
Sep 16 12:57:31.740022 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:31.739991    2577 generic.go:356] "Generic (PLEG): container finished" podID="46d533d7-d229-4daf-97de-f016c975177b" containerID="9078b80a3b7d4e3e5b45ae0097971e0f56f52872bc51da3137eaeb416e9106f7" exitCode=0
Sep 16 12:57:31.740230 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:31.740071    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d0f5a035-54c4f7fd98-hxf8s"
Sep 16 12:57:31.740230 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:31.740070    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d0f5a035-54c4f7fd98-hxf8s" event={"ID":"46d533d7-d229-4daf-97de-f016c975177b","Type":"ContainerDied","Data":"9078b80a3b7d4e3e5b45ae0097971e0f56f52872bc51da3137eaeb416e9106f7"}
Sep 16 12:57:31.740230 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:31.740129    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d0f5a035-54c4f7fd98-hxf8s" event={"ID":"46d533d7-d229-4daf-97de-f016c975177b","Type":"ContainerDied","Data":"ca6cef112639e7bf00171abf897088d6245a75007d6edf482b850bfcf99465d5"}
Sep 16 12:57:31.740230 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:31.740152    2577 scope.go:117] "RemoveContainer" containerID="9078b80a3b7d4e3e5b45ae0097971e0f56f52872bc51da3137eaeb416e9106f7"
Sep 16 12:57:31.750241 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:31.750225    2577 scope.go:117] "RemoveContainer" containerID="9078b80a3b7d4e3e5b45ae0097971e0f56f52872bc51da3137eaeb416e9106f7"
Sep 16 12:57:31.750529 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:57:31.750507    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9078b80a3b7d4e3e5b45ae0097971e0f56f52872bc51da3137eaeb416e9106f7\": container with ID starting with 9078b80a3b7d4e3e5b45ae0097971e0f56f52872bc51da3137eaeb416e9106f7 not found: ID does not exist" containerID="9078b80a3b7d4e3e5b45ae0097971e0f56f52872bc51da3137eaeb416e9106f7"
Sep 16 12:57:31.750583 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:31.750542    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9078b80a3b7d4e3e5b45ae0097971e0f56f52872bc51da3137eaeb416e9106f7"} err="failed to get container status \"9078b80a3b7d4e3e5b45ae0097971e0f56f52872bc51da3137eaeb416e9106f7\": rpc error: code = NotFound desc = could not find container \"9078b80a3b7d4e3e5b45ae0097971e0f56f52872bc51da3137eaeb416e9106f7\": container with ID starting with 9078b80a3b7d4e3e5b45ae0097971e0f56f52872bc51da3137eaeb416e9106f7 not found: ID does not exist"
Sep 16 12:57:31.763717 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:31.763697    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d0f5a035-54c4f7fd98-hxf8s"]
Sep 16 12:57:31.768978 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:31.768960    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d0f5a035-54c4f7fd98-hxf8s"]
Sep 16 12:57:32.062102 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:32.062067    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46d533d7-d229-4daf-97de-f016c975177b" path="/var/lib/kubelet/pods/46d533d7-d229-4daf-97de-f016c975177b/volumes"
Sep 16 12:57:32.062523 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:32.062502    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70b85bf6-b85a-4f41-9cba-29f2ae53377d" path="/var/lib/kubelet/pods/70b85bf6-b85a-4f41-9cba-29f2ae53377d/volumes"
Sep 16 12:57:33.290126 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:33.290084    2577 patch_prober.go:28] interesting pod/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 12:57:33.290608 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:33.290153    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf" podUID="7c6a45ec-0e42-470e-800f-b543a32fdd3b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 12:57:36.709331 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:36.709257    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"
Sep 16 12:57:36.772316 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:36.772284    2577 generic.go:356] "Generic (PLEG): container finished" podID="7c6a45ec-0e42-470e-800f-b543a32fdd3b" containerID="f55ce6d5f67143f24ecf087773640c358432185cdf9317b5e5644f5607080e2d" exitCode=0
Sep 16 12:57:36.772499 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:36.772323    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf" event={"ID":"7c6a45ec-0e42-470e-800f-b543a32fdd3b","Type":"ContainerDied","Data":"f55ce6d5f67143f24ecf087773640c358432185cdf9317b5e5644f5607080e2d"}
Sep 16 12:57:36.772499 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:36.772340    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"
Sep 16 12:57:36.772499 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:36.772390    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf" event={"ID":"7c6a45ec-0e42-470e-800f-b543a32fdd3b","Type":"ContainerDied","Data":"b8f4b28a7b6bc33fffbfe594e46f6fa350ae861837fb46d0159e07189bf4bd12"}
Sep 16 12:57:36.772499 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:36.772421    2577 scope.go:117] "RemoveContainer" containerID="f55ce6d5f67143f24ecf087773640c358432185cdf9317b5e5644f5607080e2d"
Sep 16 12:57:36.781503 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:36.781485    2577 scope.go:117] "RemoveContainer" containerID="8a8114590eed911cfe7bf100bd6522aa3c3779d9e284e600df3546acbee415a9"
Sep 16 12:57:36.791275 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:36.791250    2577 scope.go:117] "RemoveContainer" containerID="f55ce6d5f67143f24ecf087773640c358432185cdf9317b5e5644f5607080e2d"
Sep 16 12:57:36.791558 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:57:36.791536    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f55ce6d5f67143f24ecf087773640c358432185cdf9317b5e5644f5607080e2d\": container with ID starting with f55ce6d5f67143f24ecf087773640c358432185cdf9317b5e5644f5607080e2d not found: ID does not exist" containerID="f55ce6d5f67143f24ecf087773640c358432185cdf9317b5e5644f5607080e2d"
Sep 16 12:57:36.791625 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:36.791567    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f55ce6d5f67143f24ecf087773640c358432185cdf9317b5e5644f5607080e2d"} err="failed to get container status \"f55ce6d5f67143f24ecf087773640c358432185cdf9317b5e5644f5607080e2d\": rpc error: code = NotFound desc = could not find container \"f55ce6d5f67143f24ecf087773640c358432185cdf9317b5e5644f5607080e2d\": container with ID starting with f55ce6d5f67143f24ecf087773640c358432185cdf9317b5e5644f5607080e2d not found: ID does not exist"
Sep 16 12:57:36.791625 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:36.791584    2577 scope.go:117] "RemoveContainer" containerID="8a8114590eed911cfe7bf100bd6522aa3c3779d9e284e600df3546acbee415a9"
Sep 16 12:57:36.791867 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:57:36.791845    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a8114590eed911cfe7bf100bd6522aa3c3779d9e284e600df3546acbee415a9\": container with ID starting with 8a8114590eed911cfe7bf100bd6522aa3c3779d9e284e600df3546acbee415a9 not found: ID does not exist" containerID="8a8114590eed911cfe7bf100bd6522aa3c3779d9e284e600df3546acbee415a9"
Sep 16 12:57:36.791922 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:36.791876    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a8114590eed911cfe7bf100bd6522aa3c3779d9e284e600df3546acbee415a9"} err="failed to get container status \"8a8114590eed911cfe7bf100bd6522aa3c3779d9e284e600df3546acbee415a9\": rpc error: code = NotFound desc = could not find container \"8a8114590eed911cfe7bf100bd6522aa3c3779d9e284e600df3546acbee415a9\": container with ID starting with 8a8114590eed911cfe7bf100bd6522aa3c3779d9e284e600df3546acbee415a9 not found: ID does not exist"
Sep 16 12:57:36.803360 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:36.803340    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6a45ec-0e42-470e-800f-b543a32fdd3b-credential-socket\") pod \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\" (UID: \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\") "
Sep 16 12:57:36.803451 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:36.803371    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c6a45ec-0e42-470e-800f-b543a32fdd3b-istio-token\") pod \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\" (UID: \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\") "
Sep 16 12:57:36.803451 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:36.803393    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6a45ec-0e42-470e-800f-b543a32fdd3b-istio-data\") pod \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\" (UID: \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\") "
Sep 16 12:57:36.803451 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:36.803438    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6a45ec-0e42-470e-800f-b543a32fdd3b-istio-envoy\") pod \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\" (UID: \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\") "
Sep 16 12:57:36.803590 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:36.803464    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6a45ec-0e42-470e-800f-b543a32fdd3b-istiod-ca-cert\") pod \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\" (UID: \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\") "
Sep 16 12:57:36.803590 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:36.803509    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6a45ec-0e42-470e-800f-b543a32fdd3b-istio-podinfo\") pod \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\" (UID: \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\") "
Sep 16 12:57:36.803590 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:36.803564    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6a45ec-0e42-470e-800f-b543a32fdd3b-workload-certs\") pod \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\" (UID: \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\") "
Sep 16 12:57:36.803752 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:36.803601    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6a45ec-0e42-470e-800f-b543a32fdd3b-workload-socket\") pod \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\" (UID: \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\") "
Sep 16 12:57:36.803809 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:36.803596    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6a45ec-0e42-470e-800f-b543a32fdd3b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7c6a45ec-0e42-470e-800f-b543a32fdd3b" (UID: "7c6a45ec-0e42-470e-800f-b543a32fdd3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:57:36.803809 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:36.803766    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tx2ns\" (UniqueName: \"kubernetes.io/projected/7c6a45ec-0e42-470e-800f-b543a32fdd3b-kube-api-access-tx2ns\") pod \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\" (UID: \"7c6a45ec-0e42-470e-800f-b543a32fdd3b\") "
Sep 16 12:57:36.803963 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:36.803919    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6a45ec-0e42-470e-800f-b543a32fdd3b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7c6a45ec-0e42-470e-800f-b543a32fdd3b" (UID: "7c6a45ec-0e42-470e-800f-b543a32fdd3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:57:36.804056 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:36.803955    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c6a45ec-0e42-470e-800f-b543a32fdd3b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7c6a45ec-0e42-470e-800f-b543a32fdd3b" (UID: "7c6a45ec-0e42-470e-800f-b543a32fdd3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:57:36.804119 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:36.804059    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6a45ec-0e42-470e-800f-b543a32fdd3b-istio-data" (OuterVolumeSpecName: "istio-data") pod "7c6a45ec-0e42-470e-800f-b543a32fdd3b" (UID: "7c6a45ec-0e42-470e-800f-b543a32fdd3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:57:36.804364 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:36.804342    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6a45ec-0e42-470e-800f-b543a32fdd3b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7c6a45ec-0e42-470e-800f-b543a32fdd3b" (UID: "7c6a45ec-0e42-470e-800f-b543a32fdd3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:57:36.804457 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:36.804360    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6a45ec-0e42-470e-800f-b543a32fdd3b-credential-socket\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:57:36.804457 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:36.804433    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6a45ec-0e42-470e-800f-b543a32fdd3b-istio-data\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:57:36.804576 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:36.804449    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6a45ec-0e42-470e-800f-b543a32fdd3b-istiod-ca-cert\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:57:36.804576 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:36.804490    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6a45ec-0e42-470e-800f-b543a32fdd3b-workload-socket\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:57:36.806313 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:36.806287    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c6a45ec-0e42-470e-800f-b543a32fdd3b-istio-token" (OuterVolumeSpecName: "istio-token") pod "7c6a45ec-0e42-470e-800f-b543a32fdd3b" (UID: "7c6a45ec-0e42-470e-800f-b543a32fdd3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:57:36.806313 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:36.806299    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c6a45ec-0e42-470e-800f-b543a32fdd3b-kube-api-access-tx2ns" (OuterVolumeSpecName: "kube-api-access-tx2ns") pod "7c6a45ec-0e42-470e-800f-b543a32fdd3b" (UID: "7c6a45ec-0e42-470e-800f-b543a32fdd3b"). InnerVolumeSpecName "kube-api-access-tx2ns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:57:36.806568 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:36.806535    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6a45ec-0e42-470e-800f-b543a32fdd3b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7c6a45ec-0e42-470e-800f-b543a32fdd3b" (UID: "7c6a45ec-0e42-470e-800f-b543a32fdd3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:57:36.806659 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:36.806615    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c6a45ec-0e42-470e-800f-b543a32fdd3b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7c6a45ec-0e42-470e-800f-b543a32fdd3b" (UID: "7c6a45ec-0e42-470e-800f-b543a32fdd3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:57:36.905626 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:36.905595    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx2ns\" (UniqueName: \"kubernetes.io/projected/7c6a45ec-0e42-470e-800f-b543a32fdd3b-kube-api-access-tx2ns\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:57:36.905626 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:36.905622    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c6a45ec-0e42-470e-800f-b543a32fdd3b-istio-token\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:57:36.905626 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:36.905631    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6a45ec-0e42-470e-800f-b543a32fdd3b-istio-envoy\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:57:36.905840 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:36.905639    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6a45ec-0e42-470e-800f-b543a32fdd3b-istio-podinfo\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:57:36.905840 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:36.905648    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6a45ec-0e42-470e-800f-b543a32fdd3b-workload-certs\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:57:37.100713 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:37.100689    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"]
Sep 16 12:57:37.105822 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:37.105790    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-d0f5a035-openshift-default-75fc8c6448-xrwcf"]
Sep 16 12:57:37.533760 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:37.533730    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-586bab/test-model-586bab-kserve-55d4dd4689-k5q5k"]
Sep 16 12:57:37.534379 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:37.534358    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70b85bf6-b85a-4f41-9cba-29f2ae53377d" containerName="revoke-keys"
Sep 16 12:57:37.534379 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:37.534377    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="70b85bf6-b85a-4f41-9cba-29f2ae53377d" containerName="revoke-keys"
Sep 16 12:57:37.534529 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:37.534398    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c6a45ec-0e42-470e-800f-b543a32fdd3b" containerName="istio-proxy"
Sep 16 12:57:37.534529 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:37.534407    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c6a45ec-0e42-470e-800f-b543a32fdd3b" containerName="istio-proxy"
Sep 16 12:57:37.534529 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:37.534421    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46d533d7-d229-4daf-97de-f016c975177b" containerName="payload-pre-processing"
Sep 16 12:57:37.534529 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:37.534426    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="46d533d7-d229-4daf-97de-f016c975177b" containerName="payload-pre-processing"
Sep 16 12:57:37.534529 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:37.534505    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="70b85bf6-b85a-4f41-9cba-29f2ae53377d" containerName="revoke-keys"
Sep 16 12:57:37.534529 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:37.534516    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="46d533d7-d229-4daf-97de-f016c975177b" containerName="payload-pre-processing"
Sep 16 12:57:37.534529 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:37.534523    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c6a45ec-0e42-470e-800f-b543a32fdd3b" containerName="istio-proxy"
Sep 16 12:57:37.534529 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:37.534529    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c6a45ec-0e42-470e-800f-b543a32fdd3b" containerName="istio-proxy"
Sep 16 12:57:37.539194 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:37.539172    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-586bab/test-model-586bab-kserve-55d4dd4689-k5q5k"
Sep 16 12:57:37.541866 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:37.541841    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-586bab\"/\"openshift-service-ca.crt\""
Sep 16 12:57:37.543002 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:37.542978    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-586bab\"/\"kube-root-ca.crt\""
Sep 16 12:57:37.543276 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:37.543258    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-586bab\"/\"default-dockercfg-knbr9\""
Sep 16 12:57:37.543548 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:37.543528    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-586bab\"/\"test-model-586bab-kserve-self-signed-certs\""
Sep 16 12:57:37.549441 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:37.549424    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-586bab/test-model-586bab-kserve-55d4dd4689-k5q5k"]
Sep 16 12:57:37.615299 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:37.615264    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7faf6052-c278-496d-9ebc-95d90b2d193d-model-cache\") pod \"test-model-586bab-kserve-55d4dd4689-k5q5k\" (UID: \"7faf6052-c278-496d-9ebc-95d90b2d193d\") " pod="ai-tenant-e2e-inf-a-586bab/test-model-586bab-kserve-55d4dd4689-k5q5k"
Sep 16 12:57:37.615474 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:37.615308    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5x5q\" (UniqueName: \"kubernetes.io/projected/7faf6052-c278-496d-9ebc-95d90b2d193d-kube-api-access-g5x5q\") pod \"test-model-586bab-kserve-55d4dd4689-k5q5k\" (UID: \"7faf6052-c278-496d-9ebc-95d90b2d193d\") " pod="ai-tenant-e2e-inf-a-586bab/test-model-586bab-kserve-55d4dd4689-k5q5k"
Sep 16 12:57:37.615474 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:37.615354    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7faf6052-c278-496d-9ebc-95d90b2d193d-home\") pod \"test-model-586bab-kserve-55d4dd4689-k5q5k\" (UID: \"7faf6052-c278-496d-9ebc-95d90b2d193d\") " pod="ai-tenant-e2e-inf-a-586bab/test-model-586bab-kserve-55d4dd4689-k5q5k"
Sep 16 12:57:37.615474 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:37.615418    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7faf6052-c278-496d-9ebc-95d90b2d193d-dshm\") pod \"test-model-586bab-kserve-55d4dd4689-k5q5k\" (UID: \"7faf6052-c278-496d-9ebc-95d90b2d193d\") " pod="ai-tenant-e2e-inf-a-586bab/test-model-586bab-kserve-55d4dd4689-k5q5k"
Sep 16 12:57:37.615474 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:37.615464    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7faf6052-c278-496d-9ebc-95d90b2d193d-tls-certs\") pod \"test-model-586bab-kserve-55d4dd4689-k5q5k\" (UID: \"7faf6052-c278-496d-9ebc-95d90b2d193d\") " pod="ai-tenant-e2e-inf-a-586bab/test-model-586bab-kserve-55d4dd4689-k5q5k"
Sep 16 12:57:37.615625 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:37.615495    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7faf6052-c278-496d-9ebc-95d90b2d193d-tmp-dir\") pod \"test-model-586bab-kserve-55d4dd4689-k5q5k\" (UID: \"7faf6052-c278-496d-9ebc-95d90b2d193d\") " pod="ai-tenant-e2e-inf-a-586bab/test-model-586bab-kserve-55d4dd4689-k5q5k"
Sep 16 12:57:37.716406 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:37.716369    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7faf6052-c278-496d-9ebc-95d90b2d193d-dshm\") pod \"test-model-586bab-kserve-55d4dd4689-k5q5k\" (UID: \"7faf6052-c278-496d-9ebc-95d90b2d193d\") " pod="ai-tenant-e2e-inf-a-586bab/test-model-586bab-kserve-55d4dd4689-k5q5k"
Sep 16 12:57:37.716406 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:37.716412    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7faf6052-c278-496d-9ebc-95d90b2d193d-tls-certs\") pod \"test-model-586bab-kserve-55d4dd4689-k5q5k\" (UID: \"7faf6052-c278-496d-9ebc-95d90b2d193d\") " pod="ai-tenant-e2e-inf-a-586bab/test-model-586bab-kserve-55d4dd4689-k5q5k"
Sep 16 12:57:37.717001 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:37.716437    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7faf6052-c278-496d-9ebc-95d90b2d193d-tmp-dir\") pod \"test-model-586bab-kserve-55d4dd4689-k5q5k\" (UID: \"7faf6052-c278-496d-9ebc-95d90b2d193d\") " pod="ai-tenant-e2e-inf-a-586bab/test-model-586bab-kserve-55d4dd4689-k5q5k"
Sep 16 12:57:37.717001 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:37.716499    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7faf6052-c278-496d-9ebc-95d90b2d193d-model-cache\") pod \"test-model-586bab-kserve-55d4dd4689-k5q5k\" (UID: \"7faf6052-c278-496d-9ebc-95d90b2d193d\") " pod="ai-tenant-e2e-inf-a-586bab/test-model-586bab-kserve-55d4dd4689-k5q5k"
Sep 16 12:57:37.717001 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:37.716521    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5x5q\" (UniqueName: \"kubernetes.io/projected/7faf6052-c278-496d-9ebc-95d90b2d193d-kube-api-access-g5x5q\") pod \"test-model-586bab-kserve-55d4dd4689-k5q5k\" (UID: \"7faf6052-c278-496d-9ebc-95d90b2d193d\") " pod="ai-tenant-e2e-inf-a-586bab/test-model-586bab-kserve-55d4dd4689-k5q5k"
Sep 16 12:57:37.717001 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:37.716552    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7faf6052-c278-496d-9ebc-95d90b2d193d-home\") pod \"test-model-586bab-kserve-55d4dd4689-k5q5k\" (UID: \"7faf6052-c278-496d-9ebc-95d90b2d193d\") " pod="ai-tenant-e2e-inf-a-586bab/test-model-586bab-kserve-55d4dd4689-k5q5k"
Sep 16 12:57:37.717001 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:37.716838    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7faf6052-c278-496d-9ebc-95d90b2d193d-home\") pod \"test-model-586bab-kserve-55d4dd4689-k5q5k\" (UID: \"7faf6052-c278-496d-9ebc-95d90b2d193d\") " pod="ai-tenant-e2e-inf-a-586bab/test-model-586bab-kserve-55d4dd4689-k5q5k"
Sep 16 12:57:37.717001 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:37.716910    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7faf6052-c278-496d-9ebc-95d90b2d193d-tmp-dir\") pod \"test-model-586bab-kserve-55d4dd4689-k5q5k\" (UID: \"7faf6052-c278-496d-9ebc-95d90b2d193d\") " pod="ai-tenant-e2e-inf-a-586bab/test-model-586bab-kserve-55d4dd4689-k5q5k"
Sep 16 12:57:37.717001 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:37.716915    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7faf6052-c278-496d-9ebc-95d90b2d193d-model-cache\") pod \"test-model-586bab-kserve-55d4dd4689-k5q5k\" (UID: \"7faf6052-c278-496d-9ebc-95d90b2d193d\") " pod="ai-tenant-e2e-inf-a-586bab/test-model-586bab-kserve-55d4dd4689-k5q5k"
Sep 16 12:57:37.718941 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:37.718914    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7faf6052-c278-496d-9ebc-95d90b2d193d-dshm\") pod \"test-model-586bab-kserve-55d4dd4689-k5q5k\" (UID: \"7faf6052-c278-496d-9ebc-95d90b2d193d\") " pod="ai-tenant-e2e-inf-a-586bab/test-model-586bab-kserve-55d4dd4689-k5q5k"
Sep 16 12:57:37.719063 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:37.719020    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7faf6052-c278-496d-9ebc-95d90b2d193d-tls-certs\") pod \"test-model-586bab-kserve-55d4dd4689-k5q5k\" (UID: \"7faf6052-c278-496d-9ebc-95d90b2d193d\") " pod="ai-tenant-e2e-inf-a-586bab/test-model-586bab-kserve-55d4dd4689-k5q5k"
Sep 16 12:57:37.725321 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:37.725293    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5x5q\" (UniqueName: \"kubernetes.io/projected/7faf6052-c278-496d-9ebc-95d90b2d193d-kube-api-access-g5x5q\") pod \"test-model-586bab-kserve-55d4dd4689-k5q5k\" (UID: \"7faf6052-c278-496d-9ebc-95d90b2d193d\") " pod="ai-tenant-e2e-inf-a-586bab/test-model-586bab-kserve-55d4dd4689-k5q5k"
Sep 16 12:57:37.850233 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:37.850133    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-586bab/test-model-586bab-kserve-55d4dd4689-k5q5k"
Sep 16 12:57:38.056643 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:38.056607    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c6a45ec-0e42-470e-800f-b543a32fdd3b" path="/var/lib/kubelet/pods/7c6a45ec-0e42-470e-800f-b543a32fdd3b/volumes"
Sep 16 12:57:38.125902 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:38.125867    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-586bab/test-model-586bab-kserve-55d4dd4689-k5q5k"]
Sep 16 12:57:38.127836 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:57:38.127813    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7faf6052_c278_496d_9ebc_95d90b2d193d.slice/crio-27a9a95d705e880a96d8c14dac10a4ec6f00a06a6d9ae04113669bda79bc725b WatchSource:0}: Error finding container 27a9a95d705e880a96d8c14dac10a4ec6f00a06a6d9ae04113669bda79bc725b: Status 404 returned error can't find the container with id 27a9a95d705e880a96d8c14dac10a4ec6f00a06a6d9ae04113669bda79bc725b
Sep 16 12:57:38.787367 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:38.787327    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-586bab/test-model-586bab-kserve-55d4dd4689-k5q5k" event={"ID":"7faf6052-c278-496d-9ebc-95d90b2d193d","Type":"ContainerStarted","Data":"fd5a8fbadda9d11d69d5969290fe3939b29b44d1da9e31c26c3224a190d0f557"}
Sep 16 12:57:38.787367 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:38.787381    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-586bab/test-model-586bab-kserve-55d4dd4689-k5q5k" event={"ID":"7faf6052-c278-496d-9ebc-95d90b2d193d","Type":"ContainerStarted","Data":"27a9a95d705e880a96d8c14dac10a4ec6f00a06a6d9ae04113669bda79bc725b"}
Sep 16 12:57:38.805616 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:38.805568    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-586bab/test-model-586bab-kserve-55d4dd4689-k5q5k" podStartSLOduration=1.8055525989999999 podStartE2EDuration="1.805552599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:57:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:57:38.804162979 +0000 UTC m=+1165.251395622" watchObservedRunningTime="2026-09-16 12:57:38.805552599 +0000 UTC m=+1165.252785241"
Sep 16 12:57:47.391405 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:47.391367    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a24d8b-xbp6d"]
Sep 16 12:57:47.392012 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:47.391996    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c6a45ec-0e42-470e-800f-b543a32fdd3b" containerName="istio-proxy"
Sep 16 12:57:47.392086 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:47.392015    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c6a45ec-0e42-470e-800f-b543a32fdd3b" containerName="istio-proxy"
Sep 16 12:57:47.395914 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:47.395892    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a24d8b-xbp6d"
Sep 16 12:57:47.398866 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:47.398838    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:57:47.399008 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:47.398940    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-64zph\""
Sep 16 12:57:47.403687 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:47.403661    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a24d8b-xbp6d"]
Sep 16 12:57:47.515077 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:47.515015    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ef48fce-3886-4b86-bb33-0d18212b534c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-a24d8b-xbp6d\" (UID: \"4ef48fce-3886-4b86-bb33-0d18212b534c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a24d8b-xbp6d"
Sep 16 12:57:47.616255 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:47.616221    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ef48fce-3886-4b86-bb33-0d18212b534c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-a24d8b-xbp6d\" (UID: \"4ef48fce-3886-4b86-bb33-0d18212b534c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a24d8b-xbp6d"
Sep 16 12:57:47.616804 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:47.616785    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ef48fce-3886-4b86-bb33-0d18212b534c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-a24d8b-xbp6d\" (UID: \"4ef48fce-3886-4b86-bb33-0d18212b534c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a24d8b-xbp6d"
Sep 16 12:57:47.709565 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:47.709477    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a24d8b-xbp6d"
Sep 16 12:57:47.850699 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:47.850673    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-586bab/test-model-586bab-kserve-55d4dd4689-k5q5k"
Sep 16 12:57:47.850870 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:47.850715    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-586bab/test-model-586bab-kserve-55d4dd4689-k5q5k"
Sep 16 12:57:47.862921 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:47.862901    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-586bab/test-model-586bab-kserve-55d4dd4689-k5q5k"
Sep 16 12:57:47.986174 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:47.986147    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a24d8b-xbp6d"]
Sep 16 12:57:47.988645 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:57:47.988611    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ef48fce_3886_4b86_bb33_0d18212b534c.slice/crio-b03fd53e64bec17d825c1bf5ba6cc063e3561edd01dbd8ef6653c7ba05bc5030 WatchSource:0}: Error finding container b03fd53e64bec17d825c1bf5ba6cc063e3561edd01dbd8ef6653c7ba05bc5030: Status 404 returned error can't find the container with id b03fd53e64bec17d825c1bf5ba6cc063e3561edd01dbd8ef6653c7ba05bc5030
Sep 16 12:57:48.851210 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:48.851177    2577 generic.go:356] "Generic (PLEG): container finished" podID="4ef48fce-3886-4b86-bb33-0d18212b534c" containerID="bd8a8f121404aa1e858a2447f7dc0604f9d209fe0ba51ae7255234a6751af61a" exitCode=0
Sep 16 12:57:48.851639 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:48.851269    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a24d8b-xbp6d" event={"ID":"4ef48fce-3886-4b86-bb33-0d18212b534c","Type":"ContainerDied","Data":"bd8a8f121404aa1e858a2447f7dc0604f9d209fe0ba51ae7255234a6751af61a"}
Sep 16 12:57:48.851639 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:48.851325    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a24d8b-xbp6d" event={"ID":"4ef48fce-3886-4b86-bb33-0d18212b534c","Type":"ContainerStarted","Data":"b03fd53e64bec17d825c1bf5ba6cc063e3561edd01dbd8ef6653c7ba05bc5030"}
Sep 16 12:57:48.862802 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:48.862776    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-586bab/test-model-586bab-kserve-55d4dd4689-k5q5k"
Sep 16 12:57:49.999510 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:49.999486    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a24d8b-xbp6d"
Sep 16 12:57:50.142280 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:50.142201    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ef48fce-3886-4b86-bb33-0d18212b534c-maas-api-service-ca\") pod \"4ef48fce-3886-4b86-bb33-0d18212b534c\" (UID: \"4ef48fce-3886-4b86-bb33-0d18212b534c\") "
Sep 16 12:57:50.142514 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:50.142492    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ef48fce-3886-4b86-bb33-0d18212b534c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4ef48fce-3886-4b86-bb33-0d18212b534c" (UID: "4ef48fce-3886-4b86-bb33-0d18212b534c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:57:50.142578 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:50.142566    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ef48fce-3886-4b86-bb33-0d18212b534c-maas-api-service-ca\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:57:50.864979 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:50.864941    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a24d8b-xbp6d" event={"ID":"4ef48fce-3886-4b86-bb33-0d18212b534c","Type":"ContainerDied","Data":"b03fd53e64bec17d825c1bf5ba6cc063e3561edd01dbd8ef6653c7ba05bc5030"}
Sep 16 12:57:50.865173 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:50.864998    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b03fd53e64bec17d825c1bf5ba6cc063e3561edd01dbd8ef6653c7ba05bc5030"
Sep 16 12:57:50.865173 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:50.864955    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a24d8b-xbp6d"
Sep 16 12:57:51.868317 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:51.868274    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-a24d8b/autores-model-a24d8b-kserve-5878448ddc-jgxwh"]
Sep 16 12:57:58.150698 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:58.150653    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a24d8b-xbp6d"]
Sep 16 12:57:58.153186 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:57:58.153161    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a24d8b-xbp6d"]
Sep 16 12:58:00.055198 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:00.055165    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ef48fce-3886-4b86-bb33-0d18212b534c" path="/var/lib/kubelet/pods/4ef48fce-3886-4b86-bb33-0d18212b534c/volumes"
Sep 16 12:58:06.892613 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:06.892539    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-a24d8b/autores-model-a24d8b-kserve-5878448ddc-jgxwh" podUID="2a6e48b9-3e76-41c9-8c6e-b12c47c791c6" containerName="main" containerID="cri-o://adb3f94d295786c7aa1538e8270f8c9a5a830fa331affc7c7e63a83b3de33e9b" gracePeriod=45
Sep 16 12:58:07.140751 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:07.140725    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a24d8b/autores-model-a24d8b-kserve-5878448ddc-jgxwh"
Sep 16 12:58:07.227683 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:07.227587    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a6e48b9-3e76-41c9-8c6e-b12c47c791c6-home\") pod \"2a6e48b9-3e76-41c9-8c6e-b12c47c791c6\" (UID: \"2a6e48b9-3e76-41c9-8c6e-b12c47c791c6\") "
Sep 16 12:58:07.227683 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:07.227670    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a6e48b9-3e76-41c9-8c6e-b12c47c791c6-model-cache\") pod \"2a6e48b9-3e76-41c9-8c6e-b12c47c791c6\" (UID: \"2a6e48b9-3e76-41c9-8c6e-b12c47c791c6\") "
Sep 16 12:58:07.227927 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:07.227704    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a6e48b9-3e76-41c9-8c6e-b12c47c791c6-dshm\") pod \"2a6e48b9-3e76-41c9-8c6e-b12c47c791c6\" (UID: \"2a6e48b9-3e76-41c9-8c6e-b12c47c791c6\") "
Sep 16 12:58:07.227927 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:07.227759    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a6e48b9-3e76-41c9-8c6e-b12c47c791c6-tmp-dir\") pod \"2a6e48b9-3e76-41c9-8c6e-b12c47c791c6\" (UID: \"2a6e48b9-3e76-41c9-8c6e-b12c47c791c6\") "
Sep 16 12:58:07.227927 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:07.227804    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a6e48b9-3e76-41c9-8c6e-b12c47c791c6-tls-certs\") pod \"2a6e48b9-3e76-41c9-8c6e-b12c47c791c6\" (UID: \"2a6e48b9-3e76-41c9-8c6e-b12c47c791c6\") "
Sep 16 12:58:07.227927 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:07.227889    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljs5j\" (UniqueName: \"kubernetes.io/projected/2a6e48b9-3e76-41c9-8c6e-b12c47c791c6-kube-api-access-ljs5j\") pod \"2a6e48b9-3e76-41c9-8c6e-b12c47c791c6\" (UID: \"2a6e48b9-3e76-41c9-8c6e-b12c47c791c6\") "
Sep 16 12:58:07.228896 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:07.228589    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a6e48b9-3e76-41c9-8c6e-b12c47c791c6-home" (OuterVolumeSpecName: "home") pod "2a6e48b9-3e76-41c9-8c6e-b12c47c791c6" (UID: "2a6e48b9-3e76-41c9-8c6e-b12c47c791c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:58:07.228896 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:07.228601    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a6e48b9-3e76-41c9-8c6e-b12c47c791c6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2a6e48b9-3e76-41c9-8c6e-b12c47c791c6" (UID: "2a6e48b9-3e76-41c9-8c6e-b12c47c791c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:58:07.229223 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:07.229170    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a6e48b9-3e76-41c9-8c6e-b12c47c791c6-model-cache" (OuterVolumeSpecName: "model-cache") pod "2a6e48b9-3e76-41c9-8c6e-b12c47c791c6" (UID: "2a6e48b9-3e76-41c9-8c6e-b12c47c791c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:58:07.231733 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:07.231429    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a6e48b9-3e76-41c9-8c6e-b12c47c791c6-dshm" (OuterVolumeSpecName: "dshm") pod "2a6e48b9-3e76-41c9-8c6e-b12c47c791c6" (UID: "2a6e48b9-3e76-41c9-8c6e-b12c47c791c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:58:07.231733 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:07.231494    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a6e48b9-3e76-41c9-8c6e-b12c47c791c6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a6e48b9-3e76-41c9-8c6e-b12c47c791c6" (UID: "2a6e48b9-3e76-41c9-8c6e-b12c47c791c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:58:07.232174 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:07.232140    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a6e48b9-3e76-41c9-8c6e-b12c47c791c6-kube-api-access-ljs5j" (OuterVolumeSpecName: "kube-api-access-ljs5j") pod "2a6e48b9-3e76-41c9-8c6e-b12c47c791c6" (UID: "2a6e48b9-3e76-41c9-8c6e-b12c47c791c6"). InnerVolumeSpecName "kube-api-access-ljs5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:58:07.329666 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:07.329632    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljs5j\" (UniqueName: \"kubernetes.io/projected/2a6e48b9-3e76-41c9-8c6e-b12c47c791c6-kube-api-access-ljs5j\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:58:07.329666 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:07.329660    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a6e48b9-3e76-41c9-8c6e-b12c47c791c6-home\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:58:07.329666 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:07.329670    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a6e48b9-3e76-41c9-8c6e-b12c47c791c6-model-cache\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:58:07.329891 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:07.329679    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a6e48b9-3e76-41c9-8c6e-b12c47c791c6-dshm\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:58:07.329891 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:07.329686    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a6e48b9-3e76-41c9-8c6e-b12c47c791c6-tmp-dir\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:58:07.329891 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:07.329694    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a6e48b9-3e76-41c9-8c6e-b12c47c791c6-tls-certs\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:58:07.975282 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:07.975248    2577 generic.go:356] "Generic (PLEG): container finished" podID="2a6e48b9-3e76-41c9-8c6e-b12c47c791c6" containerID="adb3f94d295786c7aa1538e8270f8c9a5a830fa331affc7c7e63a83b3de33e9b" exitCode=0
Sep 16 12:58:07.975730 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:07.975291    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a24d8b/autores-model-a24d8b-kserve-5878448ddc-jgxwh"
Sep 16 12:58:07.975730 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:07.975326    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a24d8b/autores-model-a24d8b-kserve-5878448ddc-jgxwh" event={"ID":"2a6e48b9-3e76-41c9-8c6e-b12c47c791c6","Type":"ContainerDied","Data":"adb3f94d295786c7aa1538e8270f8c9a5a830fa331affc7c7e63a83b3de33e9b"}
Sep 16 12:58:07.975730 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:07.975371    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a24d8b/autores-model-a24d8b-kserve-5878448ddc-jgxwh" event={"ID":"2a6e48b9-3e76-41c9-8c6e-b12c47c791c6","Type":"ContainerDied","Data":"fdb3258a53cabe04fb9b3026d72b0e3caf07f20179f218e18106705dbd57fc2c"}
Sep 16 12:58:07.975730 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:07.975392    2577 scope.go:117] "RemoveContainer" containerID="adb3f94d295786c7aa1538e8270f8c9a5a830fa331affc7c7e63a83b3de33e9b"
Sep 16 12:58:07.985219 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:07.985202    2577 scope.go:117] "RemoveContainer" containerID="adb3f94d295786c7aa1538e8270f8c9a5a830fa331affc7c7e63a83b3de33e9b"
Sep 16 12:58:07.985470 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:58:07.985452    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adb3f94d295786c7aa1538e8270f8c9a5a830fa331affc7c7e63a83b3de33e9b\": container with ID starting with adb3f94d295786c7aa1538e8270f8c9a5a830fa331affc7c7e63a83b3de33e9b not found: ID does not exist" containerID="adb3f94d295786c7aa1538e8270f8c9a5a830fa331affc7c7e63a83b3de33e9b"
Sep 16 12:58:07.985521 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:07.985478    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adb3f94d295786c7aa1538e8270f8c9a5a830fa331affc7c7e63a83b3de33e9b"} err="failed to get container status \"adb3f94d295786c7aa1538e8270f8c9a5a830fa331affc7c7e63a83b3de33e9b\": rpc error: code = NotFound desc = could not find container \"adb3f94d295786c7aa1538e8270f8c9a5a830fa331affc7c7e63a83b3de33e9b\": container with ID starting with adb3f94d295786c7aa1538e8270f8c9a5a830fa331affc7c7e63a83b3de33e9b not found: ID does not exist"
Sep 16 12:58:08.000806 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:08.000783    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-a24d8b/autores-model-a24d8b-kserve-5878448ddc-jgxwh"]
Sep 16 12:58:08.013999 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:08.013977    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-a24d8b/autores-model-a24d8b-kserve-5878448ddc-jgxwh"]
Sep 16 12:58:08.055357 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:08.055323    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a6e48b9-3e76-41c9-8c6e-b12c47c791c6" path="/var/lib/kubelet/pods/2a6e48b9-3e76-41c9-8c6e-b12c47c791c6/volumes"
Sep 16 12:58:10.507340 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:10.507309    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"]
Sep 16 12:58:10.507798 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:10.507506    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm" podUID="1adf6c30-6569-40a5-ba22-caf26b38f80a" containerName="istio-proxy" containerID="cri-o://43f1c73b7bb3b639889ff63773e78b18431d1d194644763856a706aa5922b4c4" gracePeriod=30
Sep 16 12:58:14.110975 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:14.110941    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vbt8d_0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174/console-operator/1.log"
Sep 16 12:58:14.115207 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:14.115181    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vbt8d_0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174/console-operator/1.log"
Sep 16 12:58:15.773533 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:15.773471    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"
Sep 16 12:58:15.812535 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:15.812500    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1adf6c30-6569-40a5-ba22-caf26b38f80a-istio-podinfo\") pod \"1adf6c30-6569-40a5-ba22-caf26b38f80a\" (UID: \"1adf6c30-6569-40a5-ba22-caf26b38f80a\") "
Sep 16 12:58:15.812705 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:15.812560    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1adf6c30-6569-40a5-ba22-caf26b38f80a-istio-data\") pod \"1adf6c30-6569-40a5-ba22-caf26b38f80a\" (UID: \"1adf6c30-6569-40a5-ba22-caf26b38f80a\") "
Sep 16 12:58:15.812705 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:15.812593    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1adf6c30-6569-40a5-ba22-caf26b38f80a-workload-certs\") pod \"1adf6c30-6569-40a5-ba22-caf26b38f80a\" (UID: \"1adf6c30-6569-40a5-ba22-caf26b38f80a\") "
Sep 16 12:58:15.812705 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:15.812640    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1adf6c30-6569-40a5-ba22-caf26b38f80a-istiod-ca-cert\") pod \"1adf6c30-6569-40a5-ba22-caf26b38f80a\" (UID: \"1adf6c30-6569-40a5-ba22-caf26b38f80a\") "
Sep 16 12:58:15.812705 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:15.812670    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1adf6c30-6569-40a5-ba22-caf26b38f80a-istio-token\") pod \"1adf6c30-6569-40a5-ba22-caf26b38f80a\" (UID: \"1adf6c30-6569-40a5-ba22-caf26b38f80a\") "
Sep 16 12:58:15.812952 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:15.812710    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1adf6c30-6569-40a5-ba22-caf26b38f80a-istio-envoy\") pod \"1adf6c30-6569-40a5-ba22-caf26b38f80a\" (UID: \"1adf6c30-6569-40a5-ba22-caf26b38f80a\") "
Sep 16 12:58:15.812952 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:15.812832    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adf6c30-6569-40a5-ba22-caf26b38f80a-workload-socket\") pod \"1adf6c30-6569-40a5-ba22-caf26b38f80a\" (UID: \"1adf6c30-6569-40a5-ba22-caf26b38f80a\") "
Sep 16 12:58:15.812952 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:15.812858    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adf6c30-6569-40a5-ba22-caf26b38f80a-credential-socket\") pod \"1adf6c30-6569-40a5-ba22-caf26b38f80a\" (UID: \"1adf6c30-6569-40a5-ba22-caf26b38f80a\") "
Sep 16 12:58:15.813154 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:15.813009    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1adf6c30-6569-40a5-ba22-caf26b38f80a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1adf6c30-6569-40a5-ba22-caf26b38f80a" (UID: "1adf6c30-6569-40a5-ba22-caf26b38f80a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:58:15.813154 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:15.813022    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1adf6c30-6569-40a5-ba22-caf26b38f80a-istio-data" (OuterVolumeSpecName: "istio-data") pod "1adf6c30-6569-40a5-ba22-caf26b38f80a" (UID: "1adf6c30-6569-40a5-ba22-caf26b38f80a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:58:15.813259 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:15.813244    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1adf6c30-6569-40a5-ba22-caf26b38f80a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1adf6c30-6569-40a5-ba22-caf26b38f80a" (UID: "1adf6c30-6569-40a5-ba22-caf26b38f80a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:58:15.813343 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:15.813310    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1adf6c30-6569-40a5-ba22-caf26b38f80a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1adf6c30-6569-40a5-ba22-caf26b38f80a" (UID: "1adf6c30-6569-40a5-ba22-caf26b38f80a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:58:15.813407 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:15.813389    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77s89\" (UniqueName: \"kubernetes.io/projected/1adf6c30-6569-40a5-ba22-caf26b38f80a-kube-api-access-77s89\") pod \"1adf6c30-6569-40a5-ba22-caf26b38f80a\" (UID: \"1adf6c30-6569-40a5-ba22-caf26b38f80a\") "
Sep 16 12:58:15.813485 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:15.813459    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1adf6c30-6569-40a5-ba22-caf26b38f80a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1adf6c30-6569-40a5-ba22-caf26b38f80a" (UID: "1adf6c30-6569-40a5-ba22-caf26b38f80a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:58:15.813869 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:15.813845    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1adf6c30-6569-40a5-ba22-caf26b38f80a-istio-data\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:58:15.813972 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:15.813873    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1adf6c30-6569-40a5-ba22-caf26b38f80a-workload-certs\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:58:15.813972 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:15.813886    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1adf6c30-6569-40a5-ba22-caf26b38f80a-istiod-ca-cert\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:58:15.813972 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:15.813898    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adf6c30-6569-40a5-ba22-caf26b38f80a-workload-socket\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:58:15.813972 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:15.813911    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adf6c30-6569-40a5-ba22-caf26b38f80a-credential-socket\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:58:15.815547 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:15.815521    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1adf6c30-6569-40a5-ba22-caf26b38f80a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1adf6c30-6569-40a5-ba22-caf26b38f80a" (UID: "1adf6c30-6569-40a5-ba22-caf26b38f80a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:58:15.815763 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:15.815742    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1adf6c30-6569-40a5-ba22-caf26b38f80a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1adf6c30-6569-40a5-ba22-caf26b38f80a" (UID: "1adf6c30-6569-40a5-ba22-caf26b38f80a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:58:15.816145 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:15.816125    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1adf6c30-6569-40a5-ba22-caf26b38f80a-istio-token" (OuterVolumeSpecName: "istio-token") pod "1adf6c30-6569-40a5-ba22-caf26b38f80a" (UID: "1adf6c30-6569-40a5-ba22-caf26b38f80a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:58:15.816302 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:15.816284    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1adf6c30-6569-40a5-ba22-caf26b38f80a-kube-api-access-77s89" (OuterVolumeSpecName: "kube-api-access-77s89") pod "1adf6c30-6569-40a5-ba22-caf26b38f80a" (UID: "1adf6c30-6569-40a5-ba22-caf26b38f80a"). InnerVolumeSpecName "kube-api-access-77s89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:58:15.914921 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:15.914883    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77s89\" (UniqueName: \"kubernetes.io/projected/1adf6c30-6569-40a5-ba22-caf26b38f80a-kube-api-access-77s89\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:58:15.914921 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:15.914915    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1adf6c30-6569-40a5-ba22-caf26b38f80a-istio-podinfo\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:58:15.914921 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:15.914925    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1adf6c30-6569-40a5-ba22-caf26b38f80a-istio-token\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:58:15.915187 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:15.914933    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1adf6c30-6569-40a5-ba22-caf26b38f80a-istio-envoy\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:58:16.024426 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:16.024340    2577 generic.go:356] "Generic (PLEG): container finished" podID="1adf6c30-6569-40a5-ba22-caf26b38f80a" containerID="43f1c73b7bb3b639889ff63773e78b18431d1d194644763856a706aa5922b4c4" exitCode=0
Sep 16 12:58:16.024426 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:16.024390    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"
Sep 16 12:58:16.024623 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:16.024428    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm" event={"ID":"1adf6c30-6569-40a5-ba22-caf26b38f80a","Type":"ContainerDied","Data":"43f1c73b7bb3b639889ff63773e78b18431d1d194644763856a706aa5922b4c4"}
Sep 16 12:58:16.024623 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:16.024486    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm" event={"ID":"1adf6c30-6569-40a5-ba22-caf26b38f80a","Type":"ContainerDied","Data":"9ecfcdb3535f387764a89a741c85ecd419ec4459a40baf0fcb55faa433a7436a"}
Sep 16 12:58:16.024623 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:16.024508    2577 scope.go:117] "RemoveContainer" containerID="43f1c73b7bb3b639889ff63773e78b18431d1d194644763856a706aa5922b4c4"
Sep 16 12:58:16.034580 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:16.034562    2577 scope.go:117] "RemoveContainer" containerID="d960f8e232219bba39dd1c3f1886c89cff0d829e78db30c74539978576ab7aaf"
Sep 16 12:58:16.046326 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:16.046292    2577 scope.go:117] "RemoveContainer" containerID="43f1c73b7bb3b639889ff63773e78b18431d1d194644763856a706aa5922b4c4"
Sep 16 12:58:16.046652 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:58:16.046625    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43f1c73b7bb3b639889ff63773e78b18431d1d194644763856a706aa5922b4c4\": container with ID starting with 43f1c73b7bb3b639889ff63773e78b18431d1d194644763856a706aa5922b4c4 not found: ID does not exist" containerID="43f1c73b7bb3b639889ff63773e78b18431d1d194644763856a706aa5922b4c4"
Sep 16 12:58:16.046760 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:16.046661    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43f1c73b7bb3b639889ff63773e78b18431d1d194644763856a706aa5922b4c4"} err="failed to get container status \"43f1c73b7bb3b639889ff63773e78b18431d1d194644763856a706aa5922b4c4\": rpc error: code = NotFound desc = could not find container \"43f1c73b7bb3b639889ff63773e78b18431d1d194644763856a706aa5922b4c4\": container with ID starting with 43f1c73b7bb3b639889ff63773e78b18431d1d194644763856a706aa5922b4c4 not found: ID does not exist"
Sep 16 12:58:16.046760 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:16.046685    2577 scope.go:117] "RemoveContainer" containerID="d960f8e232219bba39dd1c3f1886c89cff0d829e78db30c74539978576ab7aaf"
Sep 16 12:58:16.046989 ip-10-0-133-190 kubenswrapper[2577]: E0916 12:58:16.046968    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d960f8e232219bba39dd1c3f1886c89cff0d829e78db30c74539978576ab7aaf\": container with ID starting with d960f8e232219bba39dd1c3f1886c89cff0d829e78db30c74539978576ab7aaf not found: ID does not exist" containerID="d960f8e232219bba39dd1c3f1886c89cff0d829e78db30c74539978576ab7aaf"
Sep 16 12:58:16.047048 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:16.046998    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d960f8e232219bba39dd1c3f1886c89cff0d829e78db30c74539978576ab7aaf"} err="failed to get container status \"d960f8e232219bba39dd1c3f1886c89cff0d829e78db30c74539978576ab7aaf\": rpc error: code = NotFound desc = could not find container \"d960f8e232219bba39dd1c3f1886c89cff0d829e78db30c74539978576ab7aaf\": container with ID starting with d960f8e232219bba39dd1c3f1886c89cff0d829e78db30c74539978576ab7aaf not found: ID does not exist"
Sep 16 12:58:16.063705 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:16.063680    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"]
Sep 16 12:58:16.067202 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:16.067182    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-a24d8b-openshift-default-84f4678f84-5fvhm"]
Sep 16 12:58:18.055104 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:18.055075    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1adf6c30-6569-40a5-ba22-caf26b38f80a" path="/var/lib/kubelet/pods/1adf6c30-6569-40a5-ba22-caf26b38f80a/volumes"
Sep 16 12:58:19.099569 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:19.099522    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-7087d2-6b96655df7-5ktj8"]
Sep 16 12:58:19.100496 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:19.100472    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1adf6c30-6569-40a5-ba22-caf26b38f80a" containerName="istio-proxy"
Sep 16 12:58:19.100604 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:19.100498    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1adf6c30-6569-40a5-ba22-caf26b38f80a" containerName="istio-proxy"
Sep 16 12:58:19.100604 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:19.100565    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a6e48b9-3e76-41c9-8c6e-b12c47c791c6" containerName="main"
Sep 16 12:58:19.100604 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:19.100579    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a6e48b9-3e76-41c9-8c6e-b12c47c791c6" containerName="main"
Sep 16 12:58:19.100604 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:19.100588    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ef48fce-3886-4b86-bb33-0d18212b534c" containerName="revoke-keys"
Sep 16 12:58:19.100604 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:19.100595    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ef48fce-3886-4b86-bb33-0d18212b534c" containerName="revoke-keys"
Sep 16 12:58:19.100844 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:19.100766    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ef48fce-3886-4b86-bb33-0d18212b534c" containerName="revoke-keys"
Sep 16 12:58:19.100844 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:19.100783    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1adf6c30-6569-40a5-ba22-caf26b38f80a" containerName="istio-proxy"
Sep 16 12:58:19.100844 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:19.100792    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a6e48b9-3e76-41c9-8c6e-b12c47c791c6" containerName="main"
Sep 16 12:58:19.106321 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:19.106301    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-7087d2-6b96655df7-5ktj8"
Sep 16 12:58:19.109375 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:19.109355    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-7087d2\""
Sep 16 12:58:19.109494 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:19.109422    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-7087d2-dockercfg-z4hst\""
Sep 16 12:58:19.115770 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:19.115747    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-7087d2-6b96655df7-5ktj8"]
Sep 16 12:58:19.249125 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:19.249091    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c88bd1ea-698b-480b-9a3f-674354738593-plugins-config-volume\") pod \"payload-processing-e2e-explicit-7087d2-6b96655df7-5ktj8\" (UID: \"c88bd1ea-698b-480b-9a3f-674354738593\") " pod="openshift-ingress/payload-processing-e2e-explicit-7087d2-6b96655df7-5ktj8"
Sep 16 12:58:19.249288 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:19.249195    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7cfd\" (UniqueName: \"kubernetes.io/projected/c88bd1ea-698b-480b-9a3f-674354738593-kube-api-access-v7cfd\") pod \"payload-processing-e2e-explicit-7087d2-6b96655df7-5ktj8\" (UID: \"c88bd1ea-698b-480b-9a3f-674354738593\") " pod="openshift-ingress/payload-processing-e2e-explicit-7087d2-6b96655df7-5ktj8"
Sep 16 12:58:19.350620 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:19.350537    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7cfd\" (UniqueName: \"kubernetes.io/projected/c88bd1ea-698b-480b-9a3f-674354738593-kube-api-access-v7cfd\") pod \"payload-processing-e2e-explicit-7087d2-6b96655df7-5ktj8\" (UID: \"c88bd1ea-698b-480b-9a3f-674354738593\") " pod="openshift-ingress/payload-processing-e2e-explicit-7087d2-6b96655df7-5ktj8"
Sep 16 12:58:19.350620 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:19.350609    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c88bd1ea-698b-480b-9a3f-674354738593-plugins-config-volume\") pod \"payload-processing-e2e-explicit-7087d2-6b96655df7-5ktj8\" (UID: \"c88bd1ea-698b-480b-9a3f-674354738593\") " pod="openshift-ingress/payload-processing-e2e-explicit-7087d2-6b96655df7-5ktj8"
Sep 16 12:58:19.351302 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:19.351285    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c88bd1ea-698b-480b-9a3f-674354738593-plugins-config-volume\") pod \"payload-processing-e2e-explicit-7087d2-6b96655df7-5ktj8\" (UID: \"c88bd1ea-698b-480b-9a3f-674354738593\") " pod="openshift-ingress/payload-processing-e2e-explicit-7087d2-6b96655df7-5ktj8"
Sep 16 12:58:19.359264 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:19.359245    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7cfd\" (UniqueName: \"kubernetes.io/projected/c88bd1ea-698b-480b-9a3f-674354738593-kube-api-access-v7cfd\") pod \"payload-processing-e2e-explicit-7087d2-6b96655df7-5ktj8\" (UID: \"c88bd1ea-698b-480b-9a3f-674354738593\") " pod="openshift-ingress/payload-processing-e2e-explicit-7087d2-6b96655df7-5ktj8"
Sep 16 12:58:19.417570 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:19.417540    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-7087d2-6b96655df7-5ktj8"
Sep 16 12:58:19.694823 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:19.694795    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-7087d2-6b96655df7-5ktj8"]
Sep 16 12:58:19.696702 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:58:19.696669    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc88bd1ea_698b_480b_9a3f_674354738593.slice/crio-3412ccc4ac7038d0ad920b353873921bd47abc9b196f4ecefe01639a5a3eb886 WatchSource:0}: Error finding container 3412ccc4ac7038d0ad920b353873921bd47abc9b196f4ecefe01639a5a3eb886: Status 404 returned error can't find the container with id 3412ccc4ac7038d0ad920b353873921bd47abc9b196f4ecefe01639a5a3eb886
Sep 16 12:58:20.054367 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:20.054333    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-7087d2-6b96655df7-5ktj8"
Sep 16 12:58:20.054542 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:20.054388    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-7087d2-6b96655df7-5ktj8" event={"ID":"c88bd1ea-698b-480b-9a3f-674354738593","Type":"ContainerStarted","Data":"df4127579d91641cea96674670e74fe18573d3c07923533834b1757325d1b73e"}
Sep 16 12:58:20.054542 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:20.054414    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-7087d2-6b96655df7-5ktj8" event={"ID":"c88bd1ea-698b-480b-9a3f-674354738593","Type":"ContainerStarted","Data":"3412ccc4ac7038d0ad920b353873921bd47abc9b196f4ecefe01639a5a3eb886"}
Sep 16 12:58:20.068206 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:20.068161    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-7087d2-6b96655df7-5ktj8" podStartSLOduration=1.068148315 podStartE2EDuration="1.068148315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:58:20.066577903 +0000 UTC m=+1206.513810560" watchObservedRunningTime="2026-09-16 12:58:20.068148315 +0000 UTC m=+1206.515380957"
Sep 16 12:58:22.335320 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:22.335286    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7d0256/premium-w4-7d0256-kserve-69b8fcc96b-bcwl4"]
Sep 16 12:58:22.335721 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:22.335708    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1adf6c30-6569-40a5-ba22-caf26b38f80a" containerName="istio-proxy"
Sep 16 12:58:22.335766 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:22.335722    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1adf6c30-6569-40a5-ba22-caf26b38f80a" containerName="istio-proxy"
Sep 16 12:58:22.335816 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:22.335806    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1adf6c30-6569-40a5-ba22-caf26b38f80a" containerName="istio-proxy"
Sep 16 12:58:22.339133 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:22.339116    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d0256/premium-w4-7d0256-kserve-69b8fcc96b-bcwl4"
Sep 16 12:58:22.343042 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:22.343008    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7d0256\"/\"premium-w4-7d0256-kserve-self-signed-certs\""
Sep 16 12:58:22.343189 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:22.343009    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-7d0256\"/\"openshift-service-ca.crt\""
Sep 16 12:58:22.343189 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:22.343008    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7d0256\"/\"default-dockercfg-8mkcq\""
Sep 16 12:58:22.343189 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:22.343008    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-7d0256\"/\"kube-root-ca.crt\""
Sep 16 12:58:22.347756 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:22.347738    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7d0256/premium-w4-7d0256-kserve-69b8fcc96b-bcwl4"]
Sep 16 12:58:22.480500 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:22.480462    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08e00b87-97af-456a-af30-10b145da3ea6-dshm\") pod \"premium-w4-7d0256-kserve-69b8fcc96b-bcwl4\" (UID: \"08e00b87-97af-456a-af30-10b145da3ea6\") " pod="e2e-models-e2e-worker-w4-7d0256/premium-w4-7d0256-kserve-69b8fcc96b-bcwl4"
Sep 16 12:58:22.480500 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:22.480504    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmp5s\" (UniqueName: \"kubernetes.io/projected/08e00b87-97af-456a-af30-10b145da3ea6-kube-api-access-rmp5s\") pod \"premium-w4-7d0256-kserve-69b8fcc96b-bcwl4\" (UID: \"08e00b87-97af-456a-af30-10b145da3ea6\") " pod="e2e-models-e2e-worker-w4-7d0256/premium-w4-7d0256-kserve-69b8fcc96b-bcwl4"
Sep 16 12:58:22.480714 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:22.480556    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08e00b87-97af-456a-af30-10b145da3ea6-model-cache\") pod \"premium-w4-7d0256-kserve-69b8fcc96b-bcwl4\" (UID: \"08e00b87-97af-456a-af30-10b145da3ea6\") " pod="e2e-models-e2e-worker-w4-7d0256/premium-w4-7d0256-kserve-69b8fcc96b-bcwl4"
Sep 16 12:58:22.480714 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:22.480592    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08e00b87-97af-456a-af30-10b145da3ea6-tmp-dir\") pod \"premium-w4-7d0256-kserve-69b8fcc96b-bcwl4\" (UID: \"08e00b87-97af-456a-af30-10b145da3ea6\") " pod="e2e-models-e2e-worker-w4-7d0256/premium-w4-7d0256-kserve-69b8fcc96b-bcwl4"
Sep 16 12:58:22.480714 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:22.480636    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08e00b87-97af-456a-af30-10b145da3ea6-home\") pod \"premium-w4-7d0256-kserve-69b8fcc96b-bcwl4\" (UID: \"08e00b87-97af-456a-af30-10b145da3ea6\") " pod="e2e-models-e2e-worker-w4-7d0256/premium-w4-7d0256-kserve-69b8fcc96b-bcwl4"
Sep 16 12:58:22.480714 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:22.480661    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08e00b87-97af-456a-af30-10b145da3ea6-tls-certs\") pod \"premium-w4-7d0256-kserve-69b8fcc96b-bcwl4\" (UID: \"08e00b87-97af-456a-af30-10b145da3ea6\") " pod="e2e-models-e2e-worker-w4-7d0256/premium-w4-7d0256-kserve-69b8fcc96b-bcwl4"
Sep 16 12:58:22.581317 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:22.581273    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08e00b87-97af-456a-af30-10b145da3ea6-model-cache\") pod \"premium-w4-7d0256-kserve-69b8fcc96b-bcwl4\" (UID: \"08e00b87-97af-456a-af30-10b145da3ea6\") " pod="e2e-models-e2e-worker-w4-7d0256/premium-w4-7d0256-kserve-69b8fcc96b-bcwl4"
Sep 16 12:58:22.581488 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:22.581352    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08e00b87-97af-456a-af30-10b145da3ea6-tmp-dir\") pod \"premium-w4-7d0256-kserve-69b8fcc96b-bcwl4\" (UID: \"08e00b87-97af-456a-af30-10b145da3ea6\") " pod="e2e-models-e2e-worker-w4-7d0256/premium-w4-7d0256-kserve-69b8fcc96b-bcwl4"
Sep 16 12:58:22.581488 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:22.581423    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08e00b87-97af-456a-af30-10b145da3ea6-home\") pod \"premium-w4-7d0256-kserve-69b8fcc96b-bcwl4\" (UID: \"08e00b87-97af-456a-af30-10b145da3ea6\") " pod="e2e-models-e2e-worker-w4-7d0256/premium-w4-7d0256-kserve-69b8fcc96b-bcwl4"
Sep 16 12:58:22.581488 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:22.581458    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08e00b87-97af-456a-af30-10b145da3ea6-tls-certs\") pod \"premium-w4-7d0256-kserve-69b8fcc96b-bcwl4\" (UID: \"08e00b87-97af-456a-af30-10b145da3ea6\") " pod="e2e-models-e2e-worker-w4-7d0256/premium-w4-7d0256-kserve-69b8fcc96b-bcwl4"
Sep 16 12:58:22.581670 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:22.581517    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08e00b87-97af-456a-af30-10b145da3ea6-dshm\") pod \"premium-w4-7d0256-kserve-69b8fcc96b-bcwl4\" (UID: \"08e00b87-97af-456a-af30-10b145da3ea6\") " pod="e2e-models-e2e-worker-w4-7d0256/premium-w4-7d0256-kserve-69b8fcc96b-bcwl4"
Sep 16 12:58:22.581670 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:22.581541    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmp5s\" (UniqueName: \"kubernetes.io/projected/08e00b87-97af-456a-af30-10b145da3ea6-kube-api-access-rmp5s\") pod \"premium-w4-7d0256-kserve-69b8fcc96b-bcwl4\" (UID: \"08e00b87-97af-456a-af30-10b145da3ea6\") " pod="e2e-models-e2e-worker-w4-7d0256/premium-w4-7d0256-kserve-69b8fcc96b-bcwl4"
Sep 16 12:58:22.581670 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:22.581658    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08e00b87-97af-456a-af30-10b145da3ea6-model-cache\") pod \"premium-w4-7d0256-kserve-69b8fcc96b-bcwl4\" (UID: \"08e00b87-97af-456a-af30-10b145da3ea6\") " pod="e2e-models-e2e-worker-w4-7d0256/premium-w4-7d0256-kserve-69b8fcc96b-bcwl4"
Sep 16 12:58:22.581841 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:22.581727    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08e00b87-97af-456a-af30-10b145da3ea6-tmp-dir\") pod \"premium-w4-7d0256-kserve-69b8fcc96b-bcwl4\" (UID: \"08e00b87-97af-456a-af30-10b145da3ea6\") " pod="e2e-models-e2e-worker-w4-7d0256/premium-w4-7d0256-kserve-69b8fcc96b-bcwl4"
Sep 16 12:58:22.581841 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:22.581827    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08e00b87-97af-456a-af30-10b145da3ea6-home\") pod \"premium-w4-7d0256-kserve-69b8fcc96b-bcwl4\" (UID: \"08e00b87-97af-456a-af30-10b145da3ea6\") " pod="e2e-models-e2e-worker-w4-7d0256/premium-w4-7d0256-kserve-69b8fcc96b-bcwl4"
Sep 16 12:58:22.584148 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:22.584126    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08e00b87-97af-456a-af30-10b145da3ea6-dshm\") pod \"premium-w4-7d0256-kserve-69b8fcc96b-bcwl4\" (UID: \"08e00b87-97af-456a-af30-10b145da3ea6\") " pod="e2e-models-e2e-worker-w4-7d0256/premium-w4-7d0256-kserve-69b8fcc96b-bcwl4"
Sep 16 12:58:22.584254 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:22.584181    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08e00b87-97af-456a-af30-10b145da3ea6-tls-certs\") pod \"premium-w4-7d0256-kserve-69b8fcc96b-bcwl4\" (UID: \"08e00b87-97af-456a-af30-10b145da3ea6\") " pod="e2e-models-e2e-worker-w4-7d0256/premium-w4-7d0256-kserve-69b8fcc96b-bcwl4"
Sep 16 12:58:22.591547 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:22.591485    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmp5s\" (UniqueName: \"kubernetes.io/projected/08e00b87-97af-456a-af30-10b145da3ea6-kube-api-access-rmp5s\") pod \"premium-w4-7d0256-kserve-69b8fcc96b-bcwl4\" (UID: \"08e00b87-97af-456a-af30-10b145da3ea6\") " pod="e2e-models-e2e-worker-w4-7d0256/premium-w4-7d0256-kserve-69b8fcc96b-bcwl4"
Sep 16 12:58:22.649515 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:22.649481    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d0256/premium-w4-7d0256-kserve-69b8fcc96b-bcwl4"
Sep 16 12:58:22.943861 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:22.943831    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7d0256/premium-w4-7d0256-kserve-69b8fcc96b-bcwl4"]
Sep 16 12:58:22.945518 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:58:22.945489    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08e00b87_97af_456a_af30_10b145da3ea6.slice/crio-f23400508fde230740972f05a656ad6fd1193d06c4838450e2c1a4cfc688b48f WatchSource:0}: Error finding container f23400508fde230740972f05a656ad6fd1193d06c4838450e2c1a4cfc688b48f: Status 404 returned error can't find the container with id f23400508fde230740972f05a656ad6fd1193d06c4838450e2c1a4cfc688b48f
Sep 16 12:58:23.069210 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:23.069173    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d0256/premium-w4-7d0256-kserve-69b8fcc96b-bcwl4" event={"ID":"08e00b87-97af-456a-af30-10b145da3ea6","Type":"ContainerStarted","Data":"8d77ab72c7fd0ea0d9eee9b530dc7d541793cf90f92e652f01344a86579a69ae"}
Sep 16 12:58:23.069376 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:23.069221    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d0256/premium-w4-7d0256-kserve-69b8fcc96b-bcwl4" event={"ID":"08e00b87-97af-456a-af30-10b145da3ea6","Type":"ContainerStarted","Data":"f23400508fde230740972f05a656ad6fd1193d06c4838450e2c1a4cfc688b48f"}
Sep 16 12:58:23.088458 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:23.088399    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7d0256/premium-w4-7d0256-kserve-69b8fcc96b-bcwl4" podStartSLOduration=1.088380047 podStartE2EDuration="1.088380047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:58:23.087709418 +0000 UTC m=+1209.534942052" watchObservedRunningTime="2026-09-16 12:58:23.088380047 +0000 UTC m=+1209.535612689"
Sep 16 12:58:31.056988 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:31.056954    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-7087d2-6b96655df7-5ktj8"
Sep 16 12:58:32.650006 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:32.649967    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7d0256/premium-w4-7d0256-kserve-69b8fcc96b-bcwl4"
Sep 16 12:58:32.650407 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:32.650061    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7d0256/premium-w4-7d0256-kserve-69b8fcc96b-bcwl4"
Sep 16 12:58:32.663195 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:32.663170    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7d0256/premium-w4-7d0256-kserve-69b8fcc96b-bcwl4"
Sep 16 12:58:33.144237 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:33.144205    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7d0256/premium-w4-7d0256-kserve-69b8fcc96b-bcwl4"
Sep 16 12:58:39.051333 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:39.051296    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-89d6bf/test-model-89d6bf-kserve-844b98bc79-xwdp2"]
Sep 16 12:58:39.058094 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:39.058067    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-89d6bf/test-model-89d6bf-kserve-844b98bc79-xwdp2"
Sep 16 12:58:39.064667 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:39.064641    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-89d6bf\"/\"openshift-service-ca.crt\""
Sep 16 12:58:39.064914 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:39.064895    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-89d6bf\"/\"kube-root-ca.crt\""
Sep 16 12:58:39.065021 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:39.064989    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-89d6bf\"/\"default-dockercfg-msnlr\""
Sep 16 12:58:39.065277 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:39.065252    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-89d6bf\"/\"test-model-89d6bf-kserve-self-signed-certs\""
Sep 16 12:58:39.069542 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:39.068686    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-89d6bf/test-model-89d6bf-kserve-844b98bc79-xwdp2"]
Sep 16 12:58:39.156638 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:39.156599    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec8e6f24-cd09-4850-a310-1e52847ee42b-home\") pod \"test-model-89d6bf-kserve-844b98bc79-xwdp2\" (UID: \"ec8e6f24-cd09-4850-a310-1e52847ee42b\") " pod="ai-tenant-e2e-inf-b-89d6bf/test-model-89d6bf-kserve-844b98bc79-xwdp2"
Sep 16 12:58:39.156638 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:39.156634    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec8e6f24-cd09-4850-a310-1e52847ee42b-model-cache\") pod \"test-model-89d6bf-kserve-844b98bc79-xwdp2\" (UID: \"ec8e6f24-cd09-4850-a310-1e52847ee42b\") " pod="ai-tenant-e2e-inf-b-89d6bf/test-model-89d6bf-kserve-844b98bc79-xwdp2"
Sep 16 12:58:39.156849 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:39.156660    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec8e6f24-cd09-4850-a310-1e52847ee42b-dshm\") pod \"test-model-89d6bf-kserve-844b98bc79-xwdp2\" (UID: \"ec8e6f24-cd09-4850-a310-1e52847ee42b\") " pod="ai-tenant-e2e-inf-b-89d6bf/test-model-89d6bf-kserve-844b98bc79-xwdp2"
Sep 16 12:58:39.156849 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:39.156776    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgvhc\" (UniqueName: \"kubernetes.io/projected/ec8e6f24-cd09-4850-a310-1e52847ee42b-kube-api-access-xgvhc\") pod \"test-model-89d6bf-kserve-844b98bc79-xwdp2\" (UID: \"ec8e6f24-cd09-4850-a310-1e52847ee42b\") " pod="ai-tenant-e2e-inf-b-89d6bf/test-model-89d6bf-kserve-844b98bc79-xwdp2"
Sep 16 12:58:39.156849 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:39.156840    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec8e6f24-cd09-4850-a310-1e52847ee42b-tls-certs\") pod \"test-model-89d6bf-kserve-844b98bc79-xwdp2\" (UID: \"ec8e6f24-cd09-4850-a310-1e52847ee42b\") " pod="ai-tenant-e2e-inf-b-89d6bf/test-model-89d6bf-kserve-844b98bc79-xwdp2"
Sep 16 12:58:39.156953 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:39.156901    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec8e6f24-cd09-4850-a310-1e52847ee42b-tmp-dir\") pod \"test-model-89d6bf-kserve-844b98bc79-xwdp2\" (UID: \"ec8e6f24-cd09-4850-a310-1e52847ee42b\") " pod="ai-tenant-e2e-inf-b-89d6bf/test-model-89d6bf-kserve-844b98bc79-xwdp2"
Sep 16 12:58:39.258114 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:39.258076    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec8e6f24-cd09-4850-a310-1e52847ee42b-tmp-dir\") pod \"test-model-89d6bf-kserve-844b98bc79-xwdp2\" (UID: \"ec8e6f24-cd09-4850-a310-1e52847ee42b\") " pod="ai-tenant-e2e-inf-b-89d6bf/test-model-89d6bf-kserve-844b98bc79-xwdp2"
Sep 16 12:58:39.258302 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:39.258122    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec8e6f24-cd09-4850-a310-1e52847ee42b-home\") pod \"test-model-89d6bf-kserve-844b98bc79-xwdp2\" (UID: \"ec8e6f24-cd09-4850-a310-1e52847ee42b\") " pod="ai-tenant-e2e-inf-b-89d6bf/test-model-89d6bf-kserve-844b98bc79-xwdp2"
Sep 16 12:58:39.258302 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:39.258147    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec8e6f24-cd09-4850-a310-1e52847ee42b-model-cache\") pod \"test-model-89d6bf-kserve-844b98bc79-xwdp2\" (UID: \"ec8e6f24-cd09-4850-a310-1e52847ee42b\") " pod="ai-tenant-e2e-inf-b-89d6bf/test-model-89d6bf-kserve-844b98bc79-xwdp2"
Sep 16 12:58:39.258302 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:39.258183    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec8e6f24-cd09-4850-a310-1e52847ee42b-dshm\") pod \"test-model-89d6bf-kserve-844b98bc79-xwdp2\" (UID: \"ec8e6f24-cd09-4850-a310-1e52847ee42b\") " pod="ai-tenant-e2e-inf-b-89d6bf/test-model-89d6bf-kserve-844b98bc79-xwdp2"
Sep 16 12:58:39.258302 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:39.258253    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgvhc\" (UniqueName: \"kubernetes.io/projected/ec8e6f24-cd09-4850-a310-1e52847ee42b-kube-api-access-xgvhc\") pod \"test-model-89d6bf-kserve-844b98bc79-xwdp2\" (UID: \"ec8e6f24-cd09-4850-a310-1e52847ee42b\") " pod="ai-tenant-e2e-inf-b-89d6bf/test-model-89d6bf-kserve-844b98bc79-xwdp2"
Sep 16 12:58:39.258525 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:39.258319    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec8e6f24-cd09-4850-a310-1e52847ee42b-tls-certs\") pod \"test-model-89d6bf-kserve-844b98bc79-xwdp2\" (UID: \"ec8e6f24-cd09-4850-a310-1e52847ee42b\") " pod="ai-tenant-e2e-inf-b-89d6bf/test-model-89d6bf-kserve-844b98bc79-xwdp2"
Sep 16 12:58:39.258584 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:39.258514    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec8e6f24-cd09-4850-a310-1e52847ee42b-home\") pod \"test-model-89d6bf-kserve-844b98bc79-xwdp2\" (UID: \"ec8e6f24-cd09-4850-a310-1e52847ee42b\") " pod="ai-tenant-e2e-inf-b-89d6bf/test-model-89d6bf-kserve-844b98bc79-xwdp2"
Sep 16 12:58:39.258584 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:39.258526    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec8e6f24-cd09-4850-a310-1e52847ee42b-tmp-dir\") pod \"test-model-89d6bf-kserve-844b98bc79-xwdp2\" (UID: \"ec8e6f24-cd09-4850-a310-1e52847ee42b\") " pod="ai-tenant-e2e-inf-b-89d6bf/test-model-89d6bf-kserve-844b98bc79-xwdp2"
Sep 16 12:58:39.258584 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:39.258576    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec8e6f24-cd09-4850-a310-1e52847ee42b-model-cache\") pod \"test-model-89d6bf-kserve-844b98bc79-xwdp2\" (UID: \"ec8e6f24-cd09-4850-a310-1e52847ee42b\") " pod="ai-tenant-e2e-inf-b-89d6bf/test-model-89d6bf-kserve-844b98bc79-xwdp2"
Sep 16 12:58:39.260581 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:39.260557    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec8e6f24-cd09-4850-a310-1e52847ee42b-dshm\") pod \"test-model-89d6bf-kserve-844b98bc79-xwdp2\" (UID: \"ec8e6f24-cd09-4850-a310-1e52847ee42b\") " pod="ai-tenant-e2e-inf-b-89d6bf/test-model-89d6bf-kserve-844b98bc79-xwdp2"
Sep 16 12:58:39.260877 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:39.260858    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec8e6f24-cd09-4850-a310-1e52847ee42b-tls-certs\") pod \"test-model-89d6bf-kserve-844b98bc79-xwdp2\" (UID: \"ec8e6f24-cd09-4850-a310-1e52847ee42b\") " pod="ai-tenant-e2e-inf-b-89d6bf/test-model-89d6bf-kserve-844b98bc79-xwdp2"
Sep 16 12:58:39.270548 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:39.270523    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgvhc\" (UniqueName: \"kubernetes.io/projected/ec8e6f24-cd09-4850-a310-1e52847ee42b-kube-api-access-xgvhc\") pod \"test-model-89d6bf-kserve-844b98bc79-xwdp2\" (UID: \"ec8e6f24-cd09-4850-a310-1e52847ee42b\") " pod="ai-tenant-e2e-inf-b-89d6bf/test-model-89d6bf-kserve-844b98bc79-xwdp2"
Sep 16 12:58:39.371984 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:39.371893    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-89d6bf/test-model-89d6bf-kserve-844b98bc79-xwdp2"
Sep 16 12:58:39.652771 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:39.652709    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-89d6bf/test-model-89d6bf-kserve-844b98bc79-xwdp2"]
Sep 16 12:58:39.655137 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:58:39.655114    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec8e6f24_cd09_4850_a310_1e52847ee42b.slice/crio-583e167306599d65e2acf6d0af5f4e7335b811ef1394e0b956aef413e745d752 WatchSource:0}: Error finding container 583e167306599d65e2acf6d0af5f4e7335b811ef1394e0b956aef413e745d752: Status 404 returned error can't find the container with id 583e167306599d65e2acf6d0af5f4e7335b811ef1394e0b956aef413e745d752
Sep 16 12:58:40.175018 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:40.174984    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-89d6bf/test-model-89d6bf-kserve-844b98bc79-xwdp2" event={"ID":"ec8e6f24-cd09-4850-a310-1e52847ee42b","Type":"ContainerStarted","Data":"2025fcad32854ceab8f538f1cf6f6137add6be93785656ce45bb83f3a45b056f"}
Sep 16 12:58:40.175430 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:40.175023    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-89d6bf/test-model-89d6bf-kserve-844b98bc79-xwdp2" event={"ID":"ec8e6f24-cd09-4850-a310-1e52847ee42b","Type":"ContainerStarted","Data":"583e167306599d65e2acf6d0af5f4e7335b811ef1394e0b956aef413e745d752"}
Sep 16 12:58:40.200296 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:40.200247    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-89d6bf/test-model-89d6bf-kserve-844b98bc79-xwdp2" podStartSLOduration=1.20023047 podStartE2EDuration="1.20023047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:58:40.197370927 +0000 UTC m=+1226.644603570" watchObservedRunningTime="2026-09-16 12:58:40.20023047 +0000 UTC m=+1226.647463112"
Sep 16 12:58:49.372015 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:49.371974    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-89d6bf/test-model-89d6bf-kserve-844b98bc79-xwdp2"
Sep 16 12:58:49.372015 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:49.372045    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-89d6bf/test-model-89d6bf-kserve-844b98bc79-xwdp2"
Sep 16 12:58:49.384678 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:49.384653    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-89d6bf/test-model-89d6bf-kserve-844b98bc79-xwdp2"
Sep 16 12:58:50.247085 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:58:50.247020    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-89d6bf/test-model-89d6bf-kserve-844b98bc79-xwdp2"
Sep 16 12:59:20.758909 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:20.758872    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e937b400-c4879cc9c-z9tnh"]
Sep 16 12:59:20.764651 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:20.764624    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e937b400-c4879cc9c-z9tnh"
Sep 16 12:59:20.767530 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:20.767499    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e937b400-dockercfg-bj8js\""
Sep 16 12:59:20.767666 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:20.767606    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e937b400\""
Sep 16 12:59:20.773522 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:20.773499    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e937b400-c4879cc9c-z9tnh"]
Sep 16 12:59:20.892841 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:20.892805    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf9f3a4e-7a04-48fe-aaad-1d6c02e557fa-plugins-config-volume\") pod \"payload-processing-e2e-mt-e937b400-c4879cc9c-z9tnh\" (UID: \"cf9f3a4e-7a04-48fe-aaad-1d6c02e557fa\") " pod="openshift-ingress/payload-processing-e2e-mt-e937b400-c4879cc9c-z9tnh"
Sep 16 12:59:20.893067 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:20.892896    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggkvh\" (UniqueName: \"kubernetes.io/projected/cf9f3a4e-7a04-48fe-aaad-1d6c02e557fa-kube-api-access-ggkvh\") pod \"payload-processing-e2e-mt-e937b400-c4879cc9c-z9tnh\" (UID: \"cf9f3a4e-7a04-48fe-aaad-1d6c02e557fa\") " pod="openshift-ingress/payload-processing-e2e-mt-e937b400-c4879cc9c-z9tnh"
Sep 16 12:59:20.993985 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:20.993953    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggkvh\" (UniqueName: \"kubernetes.io/projected/cf9f3a4e-7a04-48fe-aaad-1d6c02e557fa-kube-api-access-ggkvh\") pod \"payload-processing-e2e-mt-e937b400-c4879cc9c-z9tnh\" (UID: \"cf9f3a4e-7a04-48fe-aaad-1d6c02e557fa\") " pod="openshift-ingress/payload-processing-e2e-mt-e937b400-c4879cc9c-z9tnh"
Sep 16 12:59:20.994165 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:20.994101    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf9f3a4e-7a04-48fe-aaad-1d6c02e557fa-plugins-config-volume\") pod \"payload-processing-e2e-mt-e937b400-c4879cc9c-z9tnh\" (UID: \"cf9f3a4e-7a04-48fe-aaad-1d6c02e557fa\") " pod="openshift-ingress/payload-processing-e2e-mt-e937b400-c4879cc9c-z9tnh"
Sep 16 12:59:20.994781 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:20.994762    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf9f3a4e-7a04-48fe-aaad-1d6c02e557fa-plugins-config-volume\") pod \"payload-processing-e2e-mt-e937b400-c4879cc9c-z9tnh\" (UID: \"cf9f3a4e-7a04-48fe-aaad-1d6c02e557fa\") " pod="openshift-ingress/payload-processing-e2e-mt-e937b400-c4879cc9c-z9tnh"
Sep 16 12:59:21.002592 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:21.002572    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggkvh\" (UniqueName: \"kubernetes.io/projected/cf9f3a4e-7a04-48fe-aaad-1d6c02e557fa-kube-api-access-ggkvh\") pod \"payload-processing-e2e-mt-e937b400-c4879cc9c-z9tnh\" (UID: \"cf9f3a4e-7a04-48fe-aaad-1d6c02e557fa\") " pod="openshift-ingress/payload-processing-e2e-mt-e937b400-c4879cc9c-z9tnh"
Sep 16 12:59:21.079665 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:21.079570    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e937b400-c4879cc9c-z9tnh"
Sep 16 12:59:21.356934 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:21.356897    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e937b400-c4879cc9c-z9tnh"]
Sep 16 12:59:21.358304 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:59:21.358282    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf9f3a4e_7a04_48fe_aaad_1d6c02e557fa.slice/crio-64a2332291d7ef4cbd5f718473586493ef4d4d9a5e15181f9a48a9165186b424 WatchSource:0}: Error finding container 64a2332291d7ef4cbd5f718473586493ef4d4d9a5e15181f9a48a9165186b424: Status 404 returned error can't find the container with id 64a2332291d7ef4cbd5f718473586493ef4d4d9a5e15181f9a48a9165186b424
Sep 16 12:59:21.428149 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:21.428124    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e937b400-c4879cc9c-z9tnh" event={"ID":"cf9f3a4e-7a04-48fe-aaad-1d6c02e557fa","Type":"ContainerStarted","Data":"64a2332291d7ef4cbd5f718473586493ef4d4d9a5e15181f9a48a9165186b424"}
Sep 16 12:59:22.435600 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:22.435567    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e937b400-c4879cc9c-z9tnh" event={"ID":"cf9f3a4e-7a04-48fe-aaad-1d6c02e557fa","Type":"ContainerStarted","Data":"31aa4dc9490f01db6baaedcd4965c3221629ea5156c72a0b09203234a905dbc9"}
Sep 16 12:59:22.436121 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:22.435700    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-e937b400-c4879cc9c-z9tnh"
Sep 16 12:59:22.455790 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:22.455742    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-e937b400-c4879cc9c-z9tnh" podStartSLOduration=2.455727093 podStartE2EDuration="2.455727093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:59:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:59:22.453771215 +0000 UTC m=+1268.901003858" watchObservedRunningTime="2026-09-16 12:59:22.455727093 +0000 UTC m=+1268.902959734"
Sep 16 12:59:33.444008 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:33.443978    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-e937b400-c4879cc9c-z9tnh"
Sep 16 12:59:49.555113 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:49.555067    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-7087d2/explicit-model-7087d2-kserve-64fd59598f-55jpb"]
Sep 16 12:59:49.559247 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:49.559223    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-7087d2/explicit-model-7087d2-kserve-64fd59598f-55jpb"
Sep 16 12:59:49.562157 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:49.562135    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-7087d2\"/\"default-dockercfg-p58gn\""
Sep 16 12:59:49.562300 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:49.562280    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-7087d2\"/\"kube-root-ca.crt\""
Sep 16 12:59:49.563275 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:49.563254    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-7087d2\"/\"openshift-service-ca.crt\""
Sep 16 12:59:49.563411 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:49.563392    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-7087d2\"/\"explicit-model-7087d2-kserve-self-signed-certs\""
Sep 16 12:59:49.570221 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:49.570203    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-7087d2/explicit-model-7087d2-kserve-64fd59598f-55jpb"]
Sep 16 12:59:49.595665 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:49.595633    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lwrf\" (UniqueName: \"kubernetes.io/projected/8c646aa7-6ffe-4a92-84cc-35139428cccd-kube-api-access-6lwrf\") pod \"explicit-model-7087d2-kserve-64fd59598f-55jpb\" (UID: \"8c646aa7-6ffe-4a92-84cc-35139428cccd\") " pod="ai-tenant-e2e-explicit-7087d2/explicit-model-7087d2-kserve-64fd59598f-55jpb"
Sep 16 12:59:49.595781 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:49.595689    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c646aa7-6ffe-4a92-84cc-35139428cccd-tls-certs\") pod \"explicit-model-7087d2-kserve-64fd59598f-55jpb\" (UID: \"8c646aa7-6ffe-4a92-84cc-35139428cccd\") " pod="ai-tenant-e2e-explicit-7087d2/explicit-model-7087d2-kserve-64fd59598f-55jpb"
Sep 16 12:59:49.595781 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:49.595719    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c646aa7-6ffe-4a92-84cc-35139428cccd-home\") pod \"explicit-model-7087d2-kserve-64fd59598f-55jpb\" (UID: \"8c646aa7-6ffe-4a92-84cc-35139428cccd\") " pod="ai-tenant-e2e-explicit-7087d2/explicit-model-7087d2-kserve-64fd59598f-55jpb"
Sep 16 12:59:49.595913 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:49.595851    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c646aa7-6ffe-4a92-84cc-35139428cccd-model-cache\") pod \"explicit-model-7087d2-kserve-64fd59598f-55jpb\" (UID: \"8c646aa7-6ffe-4a92-84cc-35139428cccd\") " pod="ai-tenant-e2e-explicit-7087d2/explicit-model-7087d2-kserve-64fd59598f-55jpb"
Sep 16 12:59:49.595913 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:49.595897    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c646aa7-6ffe-4a92-84cc-35139428cccd-dshm\") pod \"explicit-model-7087d2-kserve-64fd59598f-55jpb\" (UID: \"8c646aa7-6ffe-4a92-84cc-35139428cccd\") " pod="ai-tenant-e2e-explicit-7087d2/explicit-model-7087d2-kserve-64fd59598f-55jpb"
Sep 16 12:59:49.595986 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:49.595928    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c646aa7-6ffe-4a92-84cc-35139428cccd-tmp-dir\") pod \"explicit-model-7087d2-kserve-64fd59598f-55jpb\" (UID: \"8c646aa7-6ffe-4a92-84cc-35139428cccd\") " pod="ai-tenant-e2e-explicit-7087d2/explicit-model-7087d2-kserve-64fd59598f-55jpb"
Sep 16 12:59:49.697196 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:49.697161    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c646aa7-6ffe-4a92-84cc-35139428cccd-model-cache\") pod \"explicit-model-7087d2-kserve-64fd59598f-55jpb\" (UID: \"8c646aa7-6ffe-4a92-84cc-35139428cccd\") " pod="ai-tenant-e2e-explicit-7087d2/explicit-model-7087d2-kserve-64fd59598f-55jpb"
Sep 16 12:59:49.697397 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:49.697215    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c646aa7-6ffe-4a92-84cc-35139428cccd-dshm\") pod \"explicit-model-7087d2-kserve-64fd59598f-55jpb\" (UID: \"8c646aa7-6ffe-4a92-84cc-35139428cccd\") " pod="ai-tenant-e2e-explicit-7087d2/explicit-model-7087d2-kserve-64fd59598f-55jpb"
Sep 16 12:59:49.697397 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:49.697243    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c646aa7-6ffe-4a92-84cc-35139428cccd-tmp-dir\") pod \"explicit-model-7087d2-kserve-64fd59598f-55jpb\" (UID: \"8c646aa7-6ffe-4a92-84cc-35139428cccd\") " pod="ai-tenant-e2e-explicit-7087d2/explicit-model-7087d2-kserve-64fd59598f-55jpb"
Sep 16 12:59:49.697397 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:49.697310    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lwrf\" (UniqueName: \"kubernetes.io/projected/8c646aa7-6ffe-4a92-84cc-35139428cccd-kube-api-access-6lwrf\") pod \"explicit-model-7087d2-kserve-64fd59598f-55jpb\" (UID: \"8c646aa7-6ffe-4a92-84cc-35139428cccd\") " pod="ai-tenant-e2e-explicit-7087d2/explicit-model-7087d2-kserve-64fd59598f-55jpb"
Sep 16 12:59:49.697397 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:49.697341    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c646aa7-6ffe-4a92-84cc-35139428cccd-tls-certs\") pod \"explicit-model-7087d2-kserve-64fd59598f-55jpb\" (UID: \"8c646aa7-6ffe-4a92-84cc-35139428cccd\") " pod="ai-tenant-e2e-explicit-7087d2/explicit-model-7087d2-kserve-64fd59598f-55jpb"
Sep 16 12:59:49.697397 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:49.697364    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c646aa7-6ffe-4a92-84cc-35139428cccd-home\") pod \"explicit-model-7087d2-kserve-64fd59598f-55jpb\" (UID: \"8c646aa7-6ffe-4a92-84cc-35139428cccd\") " pod="ai-tenant-e2e-explicit-7087d2/explicit-model-7087d2-kserve-64fd59598f-55jpb"
Sep 16 12:59:49.697640 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:49.697594    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c646aa7-6ffe-4a92-84cc-35139428cccd-model-cache\") pod \"explicit-model-7087d2-kserve-64fd59598f-55jpb\" (UID: \"8c646aa7-6ffe-4a92-84cc-35139428cccd\") " pod="ai-tenant-e2e-explicit-7087d2/explicit-model-7087d2-kserve-64fd59598f-55jpb"
Sep 16 12:59:49.697723 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:49.697695    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c646aa7-6ffe-4a92-84cc-35139428cccd-tmp-dir\") pod \"explicit-model-7087d2-kserve-64fd59598f-55jpb\" (UID: \"8c646aa7-6ffe-4a92-84cc-35139428cccd\") " pod="ai-tenant-e2e-explicit-7087d2/explicit-model-7087d2-kserve-64fd59598f-55jpb"
Sep 16 12:59:49.697781 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:49.697753    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c646aa7-6ffe-4a92-84cc-35139428cccd-home\") pod \"explicit-model-7087d2-kserve-64fd59598f-55jpb\" (UID: \"8c646aa7-6ffe-4a92-84cc-35139428cccd\") " pod="ai-tenant-e2e-explicit-7087d2/explicit-model-7087d2-kserve-64fd59598f-55jpb"
Sep 16 12:59:49.699616 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:49.699594    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c646aa7-6ffe-4a92-84cc-35139428cccd-dshm\") pod \"explicit-model-7087d2-kserve-64fd59598f-55jpb\" (UID: \"8c646aa7-6ffe-4a92-84cc-35139428cccd\") " pod="ai-tenant-e2e-explicit-7087d2/explicit-model-7087d2-kserve-64fd59598f-55jpb"
Sep 16 12:59:49.700237 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:49.700214    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c646aa7-6ffe-4a92-84cc-35139428cccd-tls-certs\") pod \"explicit-model-7087d2-kserve-64fd59598f-55jpb\" (UID: \"8c646aa7-6ffe-4a92-84cc-35139428cccd\") " pod="ai-tenant-e2e-explicit-7087d2/explicit-model-7087d2-kserve-64fd59598f-55jpb"
Sep 16 12:59:49.706523 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:49.706501    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lwrf\" (UniqueName: \"kubernetes.io/projected/8c646aa7-6ffe-4a92-84cc-35139428cccd-kube-api-access-6lwrf\") pod \"explicit-model-7087d2-kserve-64fd59598f-55jpb\" (UID: \"8c646aa7-6ffe-4a92-84cc-35139428cccd\") " pod="ai-tenant-e2e-explicit-7087d2/explicit-model-7087d2-kserve-64fd59598f-55jpb"
Sep 16 12:59:49.869920 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:49.869837    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-7087d2/explicit-model-7087d2-kserve-64fd59598f-55jpb"
Sep 16 12:59:50.150714 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:50.150693    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-7087d2/explicit-model-7087d2-kserve-64fd59598f-55jpb"]
Sep 16 12:59:50.152667 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:59:50.152630    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c646aa7_6ffe_4a92_84cc_35139428cccd.slice/crio-aa6db0054c8659c95d983aaaced4b2745b94cebc65f35342a479e82569740cdf WatchSource:0}: Error finding container aa6db0054c8659c95d983aaaced4b2745b94cebc65f35342a479e82569740cdf: Status 404 returned error can't find the container with id aa6db0054c8659c95d983aaaced4b2745b94cebc65f35342a479e82569740cdf
Sep 16 12:59:50.610755 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:50.610717    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-7087d2/explicit-model-7087d2-kserve-64fd59598f-55jpb" event={"ID":"8c646aa7-6ffe-4a92-84cc-35139428cccd","Type":"ContainerStarted","Data":"85c4ca5bea901a9ee1c0a766c354c84601c0e186889e82d1a12680ae5983efe6"}
Sep 16 12:59:50.611134 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:50.610769    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-7087d2/explicit-model-7087d2-kserve-64fd59598f-55jpb" event={"ID":"8c646aa7-6ffe-4a92-84cc-35139428cccd","Type":"ContainerStarted","Data":"aa6db0054c8659c95d983aaaced4b2745b94cebc65f35342a479e82569740cdf"}
Sep 16 12:59:50.633887 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:50.633839    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-7087d2/explicit-model-7087d2-kserve-64fd59598f-55jpb" podStartSLOduration=1.6338230089999999 podStartE2EDuration="1.633823009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:59:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:59:50.630867754 +0000 UTC m=+1297.078100396" watchObservedRunningTime="2026-09-16 12:59:50.633823009 +0000 UTC m=+1297.081055684"
Sep 16 12:59:54.055497 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:54.055466    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e937b400-xg8js"]
Sep 16 12:59:54.059114 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:54.059096    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e937b400-xg8js"
Sep 16 12:59:54.062011 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:54.061994    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:59:54.062168 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:54.062149    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-64zph\""
Sep 16 12:59:54.066517 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:54.066492    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e937b400-xg8js"]
Sep 16 12:59:54.143729 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:54.143691    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2c52cb-f9c0-4278-9bec-ff4fe5c568a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e937b400-xg8js\" (UID: \"1e2c52cb-f9c0-4278-9bec-ff4fe5c568a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e937b400-xg8js"
Sep 16 12:59:54.244506 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:54.244473    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2c52cb-f9c0-4278-9bec-ff4fe5c568a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e937b400-xg8js\" (UID: \"1e2c52cb-f9c0-4278-9bec-ff4fe5c568a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e937b400-xg8js"
Sep 16 12:59:54.245129 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:54.245110    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2c52cb-f9c0-4278-9bec-ff4fe5c568a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e937b400-xg8js\" (UID: \"1e2c52cb-f9c0-4278-9bec-ff4fe5c568a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e937b400-xg8js"
Sep 16 12:59:54.371590 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:54.371515    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e937b400-xg8js"
Sep 16 12:59:54.652089 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:54.652063    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e937b400-xg8js"]
Sep 16 12:59:54.654237 ip-10-0-133-190 kubenswrapper[2577]: W0916 12:59:54.654208    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e2c52cb_f9c0_4278_9bec_ff4fe5c568a9.slice/crio-d5896a9cdceb7d18b4027618e504003ecc1409990f1707612dd67a6a53336aa0 WatchSource:0}: Error finding container d5896a9cdceb7d18b4027618e504003ecc1409990f1707612dd67a6a53336aa0: Status 404 returned error can't find the container with id d5896a9cdceb7d18b4027618e504003ecc1409990f1707612dd67a6a53336aa0
Sep 16 12:59:55.645196 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:55.645163    2577 generic.go:356] "Generic (PLEG): container finished" podID="1e2c52cb-f9c0-4278-9bec-ff4fe5c568a9" containerID="49189360a08255f884c6281358e304cf716d3e3b01bc27ebbc18e19aa1bf59b2" exitCode=0
Sep 16 12:59:55.645589 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:55.645298    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e937b400-xg8js" event={"ID":"1e2c52cb-f9c0-4278-9bec-ff4fe5c568a9","Type":"ContainerDied","Data":"49189360a08255f884c6281358e304cf716d3e3b01bc27ebbc18e19aa1bf59b2"}
Sep 16 12:59:55.645589 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:55.645370    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e937b400-xg8js" event={"ID":"1e2c52cb-f9c0-4278-9bec-ff4fe5c568a9","Type":"ContainerStarted","Data":"d5896a9cdceb7d18b4027618e504003ecc1409990f1707612dd67a6a53336aa0"}
Sep 16 12:59:56.845790 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:56.845757    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e937b400-xg8js"
Sep 16 12:59:56.982440 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:56.981504    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2c52cb-f9c0-4278-9bec-ff4fe5c568a9-maas-api-service-ca\") pod \"1e2c52cb-f9c0-4278-9bec-ff4fe5c568a9\" (UID: \"1e2c52cb-f9c0-4278-9bec-ff4fe5c568a9\") "
Sep 16 12:59:56.982440 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:56.982371    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e2c52cb-f9c0-4278-9bec-ff4fe5c568a9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1e2c52cb-f9c0-4278-9bec-ff4fe5c568a9" (UID: "1e2c52cb-f9c0-4278-9bec-ff4fe5c568a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:59:57.083237 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:57.083208    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2c52cb-f9c0-4278-9bec-ff4fe5c568a9-maas-api-service-ca\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 12:59:57.658521 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:57.658491    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e937b400-xg8js"
Sep 16 12:59:57.658719 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:57.658489    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e937b400-xg8js" event={"ID":"1e2c52cb-f9c0-4278-9bec-ff4fe5c568a9","Type":"ContainerDied","Data":"d5896a9cdceb7d18b4027618e504003ecc1409990f1707612dd67a6a53336aa0"}
Sep 16 12:59:57.658719 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:57.658632    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5896a9cdceb7d18b4027618e504003ecc1409990f1707612dd67a6a53336aa0"
Sep 16 12:59:59.870441 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:59.870408    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-7087d2/explicit-model-7087d2-kserve-64fd59598f-55jpb"
Sep 16 12:59:59.870836 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:59.870465    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-7087d2/explicit-model-7087d2-kserve-64fd59598f-55jpb"
Sep 16 12:59:59.884372 ip-10-0-133-190 kubenswrapper[2577]: I0916 12:59:59.884346    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-7087d2/explicit-model-7087d2-kserve-64fd59598f-55jpb"
Sep 16 13:00:00.157160 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:00.157079    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-586bab-29826060-962qq"]
Sep 16 13:00:00.157829 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:00.157797    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e2c52cb-f9c0-4278-9bec-ff4fe5c568a9" containerName="revoke-keys"
Sep 16 13:00:00.157829 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:00.157828    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e2c52cb-f9c0-4278-9bec-ff4fe5c568a9" containerName="revoke-keys"
Sep 16 13:00:00.158053 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:00.157992    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e2c52cb-f9c0-4278-9bec-ff4fe5c568a9" containerName="revoke-keys"
Sep 16 13:00:00.162830 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:00.162807    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-89d6bf-29826060-6k7s5"]
Sep 16 13:00:00.162950 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:00.162928    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-586bab-29826060-962qq"
Sep 16 13:00:00.165802 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:00.165779    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-64zph\""
Sep 16 13:00:00.167458 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:00.167435    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e937b400-29826060-7wsr4"]
Sep 16 13:00:00.168597 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:00.168575    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-89d6bf-29826060-6k7s5"
Sep 16 13:00:00.172094 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:00.172067    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d0256-29826060-sjmst"]
Sep 16 13:00:00.172208 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:00.172195    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e937b400-29826060-7wsr4"
Sep 16 13:00:00.203810 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:00.203780    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-586bab-29826060-962qq"]
Sep 16 13:00:00.204084 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:00.204052    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-89d6bf-29826060-6k7s5"]
Sep 16 13:00:00.204392 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:00.204375    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e937b400-29826060-7wsr4"]
Sep 16 13:00:00.204534 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:00.204520    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d0256-29826060-sjmst"]
Sep 16 13:00:00.204656 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:00.204017    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d0256-29826060-sjmst"
Sep 16 13:00:00.207192 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:00.205939    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-586bab-29826060-962qq"
Sep 16 13:00:00.210955 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:00.210786    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-89d6bf-29826060-6k7s5"
Sep 16 13:00:00.253065 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:00.252543    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e937b400-29826060-7wsr4"
Sep 16 13:00:00.300235 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:00.300203    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d0256-29826060-sjmst"
Sep 16 13:00:00.563951 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:00.563825    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-89d6bf-29826060-6k7s5"]
Sep 16 13:00:00.567548 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:00.567518    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e937b400-29826060-7wsr4"]
Sep 16 13:00:00.568941 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:00:00.568909    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66fd3e38_29c6_4824_adc1_8791f3a7c002.slice/crio-4934b449f11773097e49cb3234324a838d9ffae170a46288498c706ad3ce1df4 WatchSource:0}: Error finding container 4934b449f11773097e49cb3234324a838d9ffae170a46288498c706ad3ce1df4: Status 404 returned error can't find the container with id 4934b449f11773097e49cb3234324a838d9ffae170a46288498c706ad3ce1df4
Sep 16 13:00:00.569515 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:00:00.569495    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff48e33a_933d_4ea1_a1e4_441d6fdf4627.slice/crio-735f460354eb81c3b36215b3c633740056a259dc3feb467093aa2d5615a67347 WatchSource:0}: Error finding container 735f460354eb81c3b36215b3c633740056a259dc3feb467093aa2d5615a67347: Status 404 returned error can't find the container with id 735f460354eb81c3b36215b3c633740056a259dc3feb467093aa2d5615a67347
Sep 16 13:00:00.570986 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:00:00.570964    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd778c68c_ac23_476d_97d9_8406b45b31e2.slice/crio-91b55dff70ebdb0b3cf42729f7180f8251b4d4eedb0ed58cf9b4275dd697542d WatchSource:0}: Error finding container 91b55dff70ebdb0b3cf42729f7180f8251b4d4eedb0ed58cf9b4275dd697542d: Status 404 returned error can't find the container with id 91b55dff70ebdb0b3cf42729f7180f8251b4d4eedb0ed58cf9b4275dd697542d
Sep 16 13:00:00.570986 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:00.570972    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-586bab-29826060-962qq"]
Sep 16 13:00:00.656515 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:00.656486    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d0256-29826060-sjmst"]
Sep 16 13:00:00.657943 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:00:00.657916    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd749be5d_502a_47ee_8505_edcf86455e50.slice/crio-92c2d81a22ed49aee247e1b7e1020ec0377e641b4c7cb858cf05249f8d651d2a WatchSource:0}: Error finding container 92c2d81a22ed49aee247e1b7e1020ec0377e641b4c7cb858cf05249f8d651d2a: Status 404 returned error can't find the container with id 92c2d81a22ed49aee247e1b7e1020ec0377e641b4c7cb858cf05249f8d651d2a
Sep 16 13:00:00.684549 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:00.684512    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e937b400-29826060-7wsr4" event={"ID":"ff48e33a-933d-4ea1-a1e4-441d6fdf4627","Type":"ContainerStarted","Data":"735f460354eb81c3b36215b3c633740056a259dc3feb467093aa2d5615a67347"}
Sep 16 13:00:00.686253 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:00.685893    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d0256-29826060-sjmst" event={"ID":"d749be5d-502a-47ee-8505-edcf86455e50","Type":"ContainerStarted","Data":"92c2d81a22ed49aee247e1b7e1020ec0377e641b4c7cb858cf05249f8d651d2a"}
Sep 16 13:00:00.687816 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:00.687766    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-586bab-29826060-962qq" event={"ID":"d778c68c-ac23-476d-97d9-8406b45b31e2","Type":"ContainerStarted","Data":"91b55dff70ebdb0b3cf42729f7180f8251b4d4eedb0ed58cf9b4275dd697542d"}
Sep 16 13:00:00.693333 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:00.690402    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-89d6bf-29826060-6k7s5" event={"ID":"66fd3e38-29c6-4824-adc1-8791f3a7c002","Type":"ContainerStarted","Data":"4934b449f11773097e49cb3234324a838d9ffae170a46288498c706ad3ce1df4"}
Sep 16 13:00:00.716792 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:00.716769    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-7087d2/explicit-model-7087d2-kserve-64fd59598f-55jpb"
Sep 16 13:00:01.699527 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:01.699444    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-586bab-29826060-962qq" event={"ID":"d778c68c-ac23-476d-97d9-8406b45b31e2","Type":"ContainerStarted","Data":"76266e1a952d9de5860fe275ceb935f453e9917a292286755748dca3fab9892c"}
Sep 16 13:00:01.701275 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:01.701246    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-89d6bf-29826060-6k7s5" event={"ID":"66fd3e38-29c6-4824-adc1-8791f3a7c002","Type":"ContainerStarted","Data":"f09b85ff6a528b90f92fa7735a36d89dbe2f71e1aba8fd63f82c928f8d0014e3"}
Sep 16 13:00:01.703266 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:01.703233    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e937b400-29826060-7wsr4" event={"ID":"ff48e33a-933d-4ea1-a1e4-441d6fdf4627","Type":"ContainerStarted","Data":"09bd2338ea8f03606683a5028089867bf7daf6314a08ea163f01f3e953fec78f"}
Sep 16 13:00:01.704742 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:01.704701    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d0256-29826060-sjmst" event={"ID":"d749be5d-502a-47ee-8505-edcf86455e50","Type":"ContainerStarted","Data":"14bffd5b0307c685226e090273dc887d6e0c0815bb983dcef7d986efc4b44f41"}
Sep 16 13:00:01.740298 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:01.740245    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e937b400-29826060-7wsr4" podStartSLOduration=1.74023062 podStartE2EDuration="1.74023062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:00:01.738110032 +0000 UTC m=+1308.185342676" watchObservedRunningTime="2026-09-16 13:00:01.74023062 +0000 UTC m=+1308.187463255"
Sep 16 13:00:01.740975 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:01.740942    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-586bab-29826060-962qq" podStartSLOduration=1.740935088 podStartE2EDuration="1.740935088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:00:01.723119829 +0000 UTC m=+1308.170352471" watchObservedRunningTime="2026-09-16 13:00:01.740935088 +0000 UTC m=+1308.188167730"
Sep 16 13:00:01.754299 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:01.754247    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d0256-29826060-sjmst" podStartSLOduration=1.754228952 podStartE2EDuration="1.754228952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:00:01.751614551 +0000 UTC m=+1308.198847195" watchObservedRunningTime="2026-09-16 13:00:01.754228952 +0000 UTC m=+1308.201461594"
Sep 16 13:00:01.766427 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:01.766378    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-89d6bf-29826060-6k7s5" podStartSLOduration=1.766361836 podStartE2EDuration="1.766361836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:00:01.765092123 +0000 UTC m=+1308.212324766" watchObservedRunningTime="2026-09-16 13:00:01.766361836 +0000 UTC m=+1308.213594477"
Sep 16 13:00:02.712399 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:02.712365    2577 generic.go:356] "Generic (PLEG): container finished" podID="66fd3e38-29c6-4824-adc1-8791f3a7c002" containerID="f09b85ff6a528b90f92fa7735a36d89dbe2f71e1aba8fd63f82c928f8d0014e3" exitCode=0
Sep 16 13:00:02.712825 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:02.712460    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-89d6bf-29826060-6k7s5" event={"ID":"66fd3e38-29c6-4824-adc1-8791f3a7c002","Type":"ContainerDied","Data":"f09b85ff6a528b90f92fa7735a36d89dbe2f71e1aba8fd63f82c928f8d0014e3"}
Sep 16 13:00:02.714137 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:02.714113    2577 generic.go:356] "Generic (PLEG): container finished" podID="ff48e33a-933d-4ea1-a1e4-441d6fdf4627" containerID="09bd2338ea8f03606683a5028089867bf7daf6314a08ea163f01f3e953fec78f" exitCode=0
Sep 16 13:00:02.714249 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:02.714195    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e937b400-29826060-7wsr4" event={"ID":"ff48e33a-933d-4ea1-a1e4-441d6fdf4627","Type":"ContainerDied","Data":"09bd2338ea8f03606683a5028089867bf7daf6314a08ea163f01f3e953fec78f"}
Sep 16 13:00:02.715729 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:02.715697    2577 generic.go:356] "Generic (PLEG): container finished" podID="d749be5d-502a-47ee-8505-edcf86455e50" containerID="14bffd5b0307c685226e090273dc887d6e0c0815bb983dcef7d986efc4b44f41" exitCode=0
Sep 16 13:00:02.715826 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:02.715743    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d0256-29826060-sjmst" event={"ID":"d749be5d-502a-47ee-8505-edcf86455e50","Type":"ContainerDied","Data":"14bffd5b0307c685226e090273dc887d6e0c0815bb983dcef7d986efc4b44f41"}
Sep 16 13:00:02.717318 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:02.717297    2577 generic.go:356] "Generic (PLEG): container finished" podID="d778c68c-ac23-476d-97d9-8406b45b31e2" containerID="76266e1a952d9de5860fe275ceb935f453e9917a292286755748dca3fab9892c" exitCode=0
Sep 16 13:00:02.717419 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:02.717352    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-586bab-29826060-962qq" event={"ID":"d778c68c-ac23-476d-97d9-8406b45b31e2","Type":"ContainerDied","Data":"76266e1a952d9de5860fe275ceb935f453e9917a292286755748dca3fab9892c"}
Sep 16 13:00:03.902219 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:03.902191    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-89d6bf-29826060-6k7s5"
Sep 16 13:00:03.933651 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:03.933624    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e937b400-29826060-7wsr4"
Sep 16 13:00:03.956758 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:03.956490    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-586bab-29826060-962qq"
Sep 16 13:00:03.964961 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:03.960675    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d0256-29826060-sjmst"
Sep 16 13:00:04.655587 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:04.655548    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e937b400-29826060-7wsr4"]
Sep 16 13:00:04.658769 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:04.658742    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e937b400-29826060-7wsr4"]
Sep 16 13:00:04.679559 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:04.679529    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e937b400-c4879cc9c-z9tnh"]
Sep 16 13:00:04.679857 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:04.679799    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-e937b400-c4879cc9c-z9tnh" podUID="cf9f3a4e-7a04-48fe-aaad-1d6c02e557fa" containerName="payload-processing" containerID="cri-o://31aa4dc9490f01db6baaedcd4965c3221629ea5156c72a0b09203234a905dbc9" gracePeriod=30
Sep 16 13:00:04.733159 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:04.733133    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-89d6bf-29826060-6k7s5"
Sep 16 13:00:04.733312 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:04.733152    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-89d6bf-29826060-6k7s5" event={"ID":"66fd3e38-29c6-4824-adc1-8791f3a7c002","Type":"ContainerDied","Data":"4934b449f11773097e49cb3234324a838d9ffae170a46288498c706ad3ce1df4"}
Sep 16 13:00:04.733312 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:04.733212    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4934b449f11773097e49cb3234324a838d9ffae170a46288498c706ad3ce1df4"
Sep 16 13:00:04.734512 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:04.734493    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e937b400-29826060-7wsr4"
Sep 16 13:00:04.734622 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:04.734497    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="735f460354eb81c3b36215b3c633740056a259dc3feb467093aa2d5615a67347"
Sep 16 13:00:04.735869 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:04.735855    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d0256-29826060-sjmst"
Sep 16 13:00:04.735954 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:04.735887    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d0256-29826060-sjmst" event={"ID":"d749be5d-502a-47ee-8505-edcf86455e50","Type":"ContainerDied","Data":"92c2d81a22ed49aee247e1b7e1020ec0377e641b4c7cb858cf05249f8d651d2a"}
Sep 16 13:00:04.735954 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:04.735929    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92c2d81a22ed49aee247e1b7e1020ec0377e641b4c7cb858cf05249f8d651d2a"
Sep 16 13:00:04.737170 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:04.737155    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-586bab-29826060-962qq"
Sep 16 13:00:04.737259 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:04.737185    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-586bab-29826060-962qq" event={"ID":"d778c68c-ac23-476d-97d9-8406b45b31e2","Type":"ContainerDied","Data":"91b55dff70ebdb0b3cf42729f7180f8251b4d4eedb0ed58cf9b4275dd697542d"}
Sep 16 13:00:04.737259 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:04.737221    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91b55dff70ebdb0b3cf42729f7180f8251b4d4eedb0ed58cf9b4275dd697542d"
Sep 16 13:00:04.758856 ip-10-0-133-190 kubenswrapper[2577]: E0916 13:00:04.758829    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-key-cleanup-e2e-mt-e937b400-29826060-7wsr4\" is forbidden: User \"system:node:ip-10-0-133-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-133-190.ec2.internal' and this object" podUID="ff48e33a-933d-4ea1-a1e4-441d6fdf4627" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e937b400-29826060-7wsr4"
Sep 16 13:00:04.768108 ip-10-0-133-190 kubenswrapper[2577]: E0916 13:00:04.768082    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-key-cleanup-e2e-mt-e937b400-29826060-7wsr4\" is forbidden: User \"system:node:ip-10-0-133-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-133-190.ec2.internal' and this object" podUID="ff48e33a-933d-4ea1-a1e4-441d6fdf4627" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e937b400-29826060-7wsr4"
Sep 16 13:00:04.886957 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:04.886933    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e937b400-xg8js"]
Sep 16 13:00:04.891054 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:04.890911    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e937b400-xg8js"]
Sep 16 13:00:04.938062 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:04.938018    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e937b400-c4879cc9c-z9tnh"
Sep 16 13:00:04.940722 ip-10-0-133-190 kubenswrapper[2577]: E0916 13:00:04.940693    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-key-cleanup-e2e-mt-e937b400-29826060-7wsr4\" is forbidden: User \"system:node:ip-10-0-133-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-133-190.ec2.internal' and this object" podUID="ff48e33a-933d-4ea1-a1e4-441d6fdf4627" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e937b400-29826060-7wsr4"
Sep 16 13:00:05.093546 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:05.093518    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggkvh\" (UniqueName: \"kubernetes.io/projected/cf9f3a4e-7a04-48fe-aaad-1d6c02e557fa-kube-api-access-ggkvh\") pod \"cf9f3a4e-7a04-48fe-aaad-1d6c02e557fa\" (UID: \"cf9f3a4e-7a04-48fe-aaad-1d6c02e557fa\") "
Sep 16 13:00:05.093748 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:05.093565    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf9f3a4e-7a04-48fe-aaad-1d6c02e557fa-plugins-config-volume\") pod \"cf9f3a4e-7a04-48fe-aaad-1d6c02e557fa\" (UID: \"cf9f3a4e-7a04-48fe-aaad-1d6c02e557fa\") "
Sep 16 13:00:05.094063 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:05.094006    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf9f3a4e-7a04-48fe-aaad-1d6c02e557fa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cf9f3a4e-7a04-48fe-aaad-1d6c02e557fa" (UID: "cf9f3a4e-7a04-48fe-aaad-1d6c02e557fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:00:05.095851 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:05.095829    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf9f3a4e-7a04-48fe-aaad-1d6c02e557fa-kube-api-access-ggkvh" (OuterVolumeSpecName: "kube-api-access-ggkvh") pod "cf9f3a4e-7a04-48fe-aaad-1d6c02e557fa" (UID: "cf9f3a4e-7a04-48fe-aaad-1d6c02e557fa"). InnerVolumeSpecName "kube-api-access-ggkvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:00:05.194754 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:05.194669    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggkvh\" (UniqueName: \"kubernetes.io/projected/cf9f3a4e-7a04-48fe-aaad-1d6c02e557fa-kube-api-access-ggkvh\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:00:05.194895 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:05.194752    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf9f3a4e-7a04-48fe-aaad-1d6c02e557fa-plugins-config-volume\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:00:05.744273 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:05.744238    2577 generic.go:356] "Generic (PLEG): container finished" podID="cf9f3a4e-7a04-48fe-aaad-1d6c02e557fa" containerID="31aa4dc9490f01db6baaedcd4965c3221629ea5156c72a0b09203234a905dbc9" exitCode=0
Sep 16 13:00:05.744456 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:05.744285    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e937b400-c4879cc9c-z9tnh"
Sep 16 13:00:05.744456 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:05.744335    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e937b400-c4879cc9c-z9tnh" event={"ID":"cf9f3a4e-7a04-48fe-aaad-1d6c02e557fa","Type":"ContainerDied","Data":"31aa4dc9490f01db6baaedcd4965c3221629ea5156c72a0b09203234a905dbc9"}
Sep 16 13:00:05.744456 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:05.744394    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e937b400-c4879cc9c-z9tnh" event={"ID":"cf9f3a4e-7a04-48fe-aaad-1d6c02e557fa","Type":"ContainerDied","Data":"64a2332291d7ef4cbd5f718473586493ef4d4d9a5e15181f9a48a9165186b424"}
Sep 16 13:00:05.744456 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:05.744416    2577 scope.go:117] "RemoveContainer" containerID="31aa4dc9490f01db6baaedcd4965c3221629ea5156c72a0b09203234a905dbc9"
Sep 16 13:00:05.747094 ip-10-0-133-190 kubenswrapper[2577]: E0916 13:00:05.747065    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-key-cleanup-e2e-mt-e937b400-29826060-7wsr4\" is forbidden: User \"system:node:ip-10-0-133-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-133-190.ec2.internal' and this object" podUID="ff48e33a-933d-4ea1-a1e4-441d6fdf4627" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e937b400-29826060-7wsr4"
Sep 16 13:00:05.754564 ip-10-0-133-190 kubenswrapper[2577]: E0916 13:00:05.754541    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-key-cleanup-e2e-mt-e937b400-29826060-7wsr4\" is forbidden: User \"system:node:ip-10-0-133-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-133-190.ec2.internal' and this object" podUID="ff48e33a-933d-4ea1-a1e4-441d6fdf4627" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e937b400-29826060-7wsr4"
Sep 16 13:00:05.754640 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:05.754577    2577 scope.go:117] "RemoveContainer" containerID="31aa4dc9490f01db6baaedcd4965c3221629ea5156c72a0b09203234a905dbc9"
Sep 16 13:00:05.754885 ip-10-0-133-190 kubenswrapper[2577]: E0916 13:00:05.754866    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31aa4dc9490f01db6baaedcd4965c3221629ea5156c72a0b09203234a905dbc9\": container with ID starting with 31aa4dc9490f01db6baaedcd4965c3221629ea5156c72a0b09203234a905dbc9 not found: ID does not exist" containerID="31aa4dc9490f01db6baaedcd4965c3221629ea5156c72a0b09203234a905dbc9"
Sep 16 13:00:05.754930 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:05.754893    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31aa4dc9490f01db6baaedcd4965c3221629ea5156c72a0b09203234a905dbc9"} err="failed to get container status \"31aa4dc9490f01db6baaedcd4965c3221629ea5156c72a0b09203234a905dbc9\": rpc error: code = NotFound desc = could not find container \"31aa4dc9490f01db6baaedcd4965c3221629ea5156c72a0b09203234a905dbc9\": container with ID starting with 31aa4dc9490f01db6baaedcd4965c3221629ea5156c72a0b09203234a905dbc9 not found: ID does not exist"
Sep 16 13:00:05.768874 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:05.768850    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e937b400-c4879cc9c-z9tnh"]
Sep 16 13:00:05.774635 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:05.774612    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e937b400-c4879cc9c-z9tnh"]
Sep 16 13:00:06.058479 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:06.058441    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e2c52cb-f9c0-4278-9bec-ff4fe5c568a9" path="/var/lib/kubelet/pods/1e2c52cb-f9c0-4278-9bec-ff4fe5c568a9/volumes"
Sep 16 13:00:06.059598 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:06.059572    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf9f3a4e-7a04-48fe-aaad-1d6c02e557fa" path="/var/lib/kubelet/pods/cf9f3a4e-7a04-48fe-aaad-1d6c02e557fa/volumes"
Sep 16 13:00:06.060211 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:06.060192    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff48e33a-933d-4ea1-a1e4-441d6fdf4627" path="/var/lib/kubelet/pods/ff48e33a-933d-4ea1-a1e4-441d6fdf4627/volumes"
Sep 16 13:00:22.466834 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:22.466791    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7087d2-bvgw8"]
Sep 16 13:00:22.467948 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:22.467923    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d749be5d-502a-47ee-8505-edcf86455e50" containerName="cleanup"
Sep 16 13:00:22.468086 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:22.467950    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d749be5d-502a-47ee-8505-edcf86455e50" containerName="cleanup"
Sep 16 13:00:22.468086 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:22.467977    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66fd3e38-29c6-4824-adc1-8791f3a7c002" containerName="cleanup"
Sep 16 13:00:22.468086 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:22.467985    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="66fd3e38-29c6-4824-adc1-8791f3a7c002" containerName="cleanup"
Sep 16 13:00:22.468086 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:22.468012    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d778c68c-ac23-476d-97d9-8406b45b31e2" containerName="cleanup"
Sep 16 13:00:22.468086 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:22.468020    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d778c68c-ac23-476d-97d9-8406b45b31e2" containerName="cleanup"
Sep 16 13:00:22.468086 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:22.468062    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf9f3a4e-7a04-48fe-aaad-1d6c02e557fa" containerName="payload-processing"
Sep 16 13:00:22.468086 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:22.468070    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf9f3a4e-7a04-48fe-aaad-1d6c02e557fa" containerName="payload-processing"
Sep 16 13:00:22.468086 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:22.468087    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff48e33a-933d-4ea1-a1e4-441d6fdf4627" containerName="cleanup"
Sep 16 13:00:22.468463 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:22.468095    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff48e33a-933d-4ea1-a1e4-441d6fdf4627" containerName="cleanup"
Sep 16 13:00:22.468463 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:22.468296    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff48e33a-933d-4ea1-a1e4-441d6fdf4627" containerName="cleanup"
Sep 16 13:00:22.468463 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:22.468316    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d749be5d-502a-47ee-8505-edcf86455e50" containerName="cleanup"
Sep 16 13:00:22.468463 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:22.468327    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d778c68c-ac23-476d-97d9-8406b45b31e2" containerName="cleanup"
Sep 16 13:00:22.468463 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:22.468347    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf9f3a4e-7a04-48fe-aaad-1d6c02e557fa" containerName="payload-processing"
Sep 16 13:00:22.468463 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:22.468364    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="66fd3e38-29c6-4824-adc1-8791f3a7c002" containerName="cleanup"
Sep 16 13:00:22.476106 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:22.476073    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7087d2-bvgw8"
Sep 16 13:00:22.478470 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:22.478442    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7087d2-bvgw8"]
Sep 16 13:00:22.479061 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:22.479043    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-64zph\""
Sep 16 13:00:22.479061 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:22.479051    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 13:00:22.572079 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:22.572016    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c05b6985-688b-4730-ab03-bd11f2b42a6c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-7087d2-bvgw8\" (UID: \"c05b6985-688b-4730-ab03-bd11f2b42a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7087d2-bvgw8"
Sep 16 13:00:22.672921 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:22.672879    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c05b6985-688b-4730-ab03-bd11f2b42a6c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-7087d2-bvgw8\" (UID: \"c05b6985-688b-4730-ab03-bd11f2b42a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7087d2-bvgw8"
Sep 16 13:00:22.673526 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:22.673507    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c05b6985-688b-4730-ab03-bd11f2b42a6c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-7087d2-bvgw8\" (UID: \"c05b6985-688b-4730-ab03-bd11f2b42a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7087d2-bvgw8"
Sep 16 13:00:22.788888 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:22.788852    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7087d2-bvgw8"
Sep 16 13:00:23.072078 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:23.072050    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7087d2-bvgw8"]
Sep 16 13:00:23.073584 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:00:23.073556    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc05b6985_688b_4730_ab03_bd11f2b42a6c.slice/crio-0654dd7ab3776ec644001ed1ff7c15113c75452a055db5b5385d9bd099bc3713 WatchSource:0}: Error finding container 0654dd7ab3776ec644001ed1ff7c15113c75452a055db5b5385d9bd099bc3713: Status 404 returned error can't find the container with id 0654dd7ab3776ec644001ed1ff7c15113c75452a055db5b5385d9bd099bc3713
Sep 16 13:00:23.873610 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:23.873570    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7087d2-bvgw8" event={"ID":"c05b6985-688b-4730-ab03-bd11f2b42a6c","Type":"ContainerStarted","Data":"069b37734d2e1aef01e6925cc7d70d4ef0fa8f9274e04d2724d038c2ea40c7a1"}
Sep 16 13:00:23.873992 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:23.873633    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7087d2-bvgw8" event={"ID":"c05b6985-688b-4730-ab03-bd11f2b42a6c","Type":"ContainerStarted","Data":"0654dd7ab3776ec644001ed1ff7c15113c75452a055db5b5385d9bd099bc3713"}
Sep 16 13:00:23.893630 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:23.893580    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7087d2-bvgw8" podStartSLOduration=1.893565758 podStartE2EDuration="1.893565758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:00:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:00:23.891895614 +0000 UTC m=+1330.339128257" watchObservedRunningTime="2026-09-16 13:00:23.893565758 +0000 UTC m=+1330.340798400"
Sep 16 13:00:24.881506 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:24.881466    2577 generic.go:356] "Generic (PLEG): container finished" podID="c05b6985-688b-4730-ab03-bd11f2b42a6c" containerID="069b37734d2e1aef01e6925cc7d70d4ef0fa8f9274e04d2724d038c2ea40c7a1" exitCode=0
Sep 16 13:00:24.881890 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:24.881562    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7087d2-bvgw8" event={"ID":"c05b6985-688b-4730-ab03-bd11f2b42a6c","Type":"ContainerDied","Data":"069b37734d2e1aef01e6925cc7d70d4ef0fa8f9274e04d2724d038c2ea40c7a1"}
Sep 16 13:00:26.044216 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:26.044191    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7087d2-bvgw8"
Sep 16 13:00:26.112450 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:26.112420    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c05b6985-688b-4730-ab03-bd11f2b42a6c-maas-api-service-ca\") pod \"c05b6985-688b-4730-ab03-bd11f2b42a6c\" (UID: \"c05b6985-688b-4730-ab03-bd11f2b42a6c\") "
Sep 16 13:00:26.112830 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:26.112808    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c05b6985-688b-4730-ab03-bd11f2b42a6c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c05b6985-688b-4730-ab03-bd11f2b42a6c" (UID: "c05b6985-688b-4730-ab03-bd11f2b42a6c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:00:26.113094 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:26.113070    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c05b6985-688b-4730-ab03-bd11f2b42a6c-maas-api-service-ca\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:00:26.543908 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:26.543872    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-7087d2/explicit-model-7087d2-kserve-64fd59598f-55jpb"]
Sep 16 13:00:26.896233 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:26.896152    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7087d2-bvgw8"
Sep 16 13:00:26.896233 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:26.896180    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7087d2-bvgw8" event={"ID":"c05b6985-688b-4730-ab03-bd11f2b42a6c","Type":"ContainerDied","Data":"0654dd7ab3776ec644001ed1ff7c15113c75452a055db5b5385d9bd099bc3713"}
Sep 16 13:00:26.896419 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:26.896238    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0654dd7ab3776ec644001ed1ff7c15113c75452a055db5b5385d9bd099bc3713"
Sep 16 13:00:36.419464 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:36.419362    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-7087d2-6b96655df7-5ktj8"]
Sep 16 13:00:36.419905 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:36.419617    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-7087d2-6b96655df7-5ktj8" podUID="c88bd1ea-698b-480b-9a3f-674354738593" containerName="payload-processing" containerID="cri-o://df4127579d91641cea96674670e74fe18573d3c07923533834b1757325d1b73e" gracePeriod=30
Sep 16 13:00:36.679772 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:36.679705    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-7087d2-6b96655df7-5ktj8"
Sep 16 13:00:36.837177 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:36.837148    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7cfd\" (UniqueName: \"kubernetes.io/projected/c88bd1ea-698b-480b-9a3f-674354738593-kube-api-access-v7cfd\") pod \"c88bd1ea-698b-480b-9a3f-674354738593\" (UID: \"c88bd1ea-698b-480b-9a3f-674354738593\") "
Sep 16 13:00:36.837389 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:36.837191    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c88bd1ea-698b-480b-9a3f-674354738593-plugins-config-volume\") pod \"c88bd1ea-698b-480b-9a3f-674354738593\" (UID: \"c88bd1ea-698b-480b-9a3f-674354738593\") "
Sep 16 13:00:36.837574 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:36.837549    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c88bd1ea-698b-480b-9a3f-674354738593-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c88bd1ea-698b-480b-9a3f-674354738593" (UID: "c88bd1ea-698b-480b-9a3f-674354738593"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:00:36.839718 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:36.839683    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c88bd1ea-698b-480b-9a3f-674354738593-kube-api-access-v7cfd" (OuterVolumeSpecName: "kube-api-access-v7cfd") pod "c88bd1ea-698b-480b-9a3f-674354738593" (UID: "c88bd1ea-698b-480b-9a3f-674354738593"). InnerVolumeSpecName "kube-api-access-v7cfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:00:36.854171 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:36.854141    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7087d2-bvgw8"]
Sep 16 13:00:36.856623 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:36.856597    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7087d2-bvgw8"]
Sep 16 13:00:36.938996 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:36.938916    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7cfd\" (UniqueName: \"kubernetes.io/projected/c88bd1ea-698b-480b-9a3f-674354738593-kube-api-access-v7cfd\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:00:36.938996 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:36.938942    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c88bd1ea-698b-480b-9a3f-674354738593-plugins-config-volume\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:00:36.958455 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:36.958425    2577 generic.go:356] "Generic (PLEG): container finished" podID="c88bd1ea-698b-480b-9a3f-674354738593" containerID="df4127579d91641cea96674670e74fe18573d3c07923533834b1757325d1b73e" exitCode=0
Sep 16 13:00:36.958626 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:36.958474    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-7087d2-6b96655df7-5ktj8"
Sep 16 13:00:36.958626 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:36.958485    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-7087d2-6b96655df7-5ktj8" event={"ID":"c88bd1ea-698b-480b-9a3f-674354738593","Type":"ContainerDied","Data":"df4127579d91641cea96674670e74fe18573d3c07923533834b1757325d1b73e"}
Sep 16 13:00:36.958626 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:36.958529    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-7087d2-6b96655df7-5ktj8" event={"ID":"c88bd1ea-698b-480b-9a3f-674354738593","Type":"ContainerDied","Data":"3412ccc4ac7038d0ad920b353873921bd47abc9b196f4ecefe01639a5a3eb886"}
Sep 16 13:00:36.958626 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:36.958549    2577 scope.go:117] "RemoveContainer" containerID="df4127579d91641cea96674670e74fe18573d3c07923533834b1757325d1b73e"
Sep 16 13:00:36.968363 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:36.968346    2577 scope.go:117] "RemoveContainer" containerID="df4127579d91641cea96674670e74fe18573d3c07923533834b1757325d1b73e"
Sep 16 13:00:36.968639 ip-10-0-133-190 kubenswrapper[2577]: E0916 13:00:36.968619    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df4127579d91641cea96674670e74fe18573d3c07923533834b1757325d1b73e\": container with ID starting with df4127579d91641cea96674670e74fe18573d3c07923533834b1757325d1b73e not found: ID does not exist" containerID="df4127579d91641cea96674670e74fe18573d3c07923533834b1757325d1b73e"
Sep 16 13:00:36.968703 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:36.968647    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df4127579d91641cea96674670e74fe18573d3c07923533834b1757325d1b73e"} err="failed to get container status \"df4127579d91641cea96674670e74fe18573d3c07923533834b1757325d1b73e\": rpc error: code = NotFound desc = could not find container \"df4127579d91641cea96674670e74fe18573d3c07923533834b1757325d1b73e\": container with ID starting with df4127579d91641cea96674670e74fe18573d3c07923533834b1757325d1b73e not found: ID does not exist"
Sep 16 13:00:36.985842 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:36.985823    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-7087d2-6b96655df7-5ktj8"]
Sep 16 13:00:36.995236 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:36.995210    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-7087d2-6b96655df7-5ktj8"]
Sep 16 13:00:38.055985 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:38.055954    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c05b6985-688b-4730-ab03-bd11f2b42a6c" path="/var/lib/kubelet/pods/c05b6985-688b-4730-ab03-bd11f2b42a6c/volumes"
Sep 16 13:00:38.056359 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:38.056337    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c88bd1ea-698b-480b-9a3f-674354738593" path="/var/lib/kubelet/pods/c88bd1ea-698b-480b-9a3f-674354738593/volumes"
Sep 16 13:00:41.566739 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:41.566691    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-7087d2/explicit-model-7087d2-kserve-64fd59598f-55jpb" podUID="8c646aa7-6ffe-4a92-84cc-35139428cccd" containerName="main" containerID="cri-o://85c4ca5bea901a9ee1c0a766c354c84601c0e186889e82d1a12680ae5983efe6" gracePeriod=45
Sep 16 13:00:41.836410 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:41.836387    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-7087d2/explicit-model-7087d2-kserve-64fd59598f-55jpb"
Sep 16 13:00:41.999909 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:41.999870    2577 generic.go:356] "Generic (PLEG): container finished" podID="8c646aa7-6ffe-4a92-84cc-35139428cccd" containerID="85c4ca5bea901a9ee1c0a766c354c84601c0e186889e82d1a12680ae5983efe6" exitCode=0
Sep 16 13:00:42.000117 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:41.999921    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-7087d2/explicit-model-7087d2-kserve-64fd59598f-55jpb"
Sep 16 13:00:42.000117 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:41.999958    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-7087d2/explicit-model-7087d2-kserve-64fd59598f-55jpb" event={"ID":"8c646aa7-6ffe-4a92-84cc-35139428cccd","Type":"ContainerDied","Data":"85c4ca5bea901a9ee1c0a766c354c84601c0e186889e82d1a12680ae5983efe6"}
Sep 16 13:00:42.000117 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:42.000008    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-7087d2/explicit-model-7087d2-kserve-64fd59598f-55jpb" event={"ID":"8c646aa7-6ffe-4a92-84cc-35139428cccd","Type":"ContainerDied","Data":"aa6db0054c8659c95d983aaaced4b2745b94cebc65f35342a479e82569740cdf"}
Sep 16 13:00:42.000117 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:42.000068    2577 scope.go:117] "RemoveContainer" containerID="85c4ca5bea901a9ee1c0a766c354c84601c0e186889e82d1a12680ae5983efe6"
Sep 16 13:00:42.000591 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:42.000570    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lwrf\" (UniqueName: \"kubernetes.io/projected/8c646aa7-6ffe-4a92-84cc-35139428cccd-kube-api-access-6lwrf\") pod \"8c646aa7-6ffe-4a92-84cc-35139428cccd\" (UID: \"8c646aa7-6ffe-4a92-84cc-35139428cccd\") "
Sep 16 13:00:42.000679 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:42.000609    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c646aa7-6ffe-4a92-84cc-35139428cccd-home\") pod \"8c646aa7-6ffe-4a92-84cc-35139428cccd\" (UID: \"8c646aa7-6ffe-4a92-84cc-35139428cccd\") "
Sep 16 13:00:42.000679 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:42.000670    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c646aa7-6ffe-4a92-84cc-35139428cccd-dshm\") pod \"8c646aa7-6ffe-4a92-84cc-35139428cccd\" (UID: \"8c646aa7-6ffe-4a92-84cc-35139428cccd\") "
Sep 16 13:00:42.000787 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:42.000702    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c646aa7-6ffe-4a92-84cc-35139428cccd-tls-certs\") pod \"8c646aa7-6ffe-4a92-84cc-35139428cccd\" (UID: \"8c646aa7-6ffe-4a92-84cc-35139428cccd\") "
Sep 16 13:00:42.000787 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:42.000756    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c646aa7-6ffe-4a92-84cc-35139428cccd-model-cache\") pod \"8c646aa7-6ffe-4a92-84cc-35139428cccd\" (UID: \"8c646aa7-6ffe-4a92-84cc-35139428cccd\") "
Sep 16 13:00:42.000889 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:42.000815    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c646aa7-6ffe-4a92-84cc-35139428cccd-tmp-dir\") pod \"8c646aa7-6ffe-4a92-84cc-35139428cccd\" (UID: \"8c646aa7-6ffe-4a92-84cc-35139428cccd\") "
Sep 16 13:00:42.000941 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:42.000879    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c646aa7-6ffe-4a92-84cc-35139428cccd-home" (OuterVolumeSpecName: "home") pod "8c646aa7-6ffe-4a92-84cc-35139428cccd" (UID: "8c646aa7-6ffe-4a92-84cc-35139428cccd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:00:42.001308 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:42.001245    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c646aa7-6ffe-4a92-84cc-35139428cccd-home\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:00:42.001430 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:42.001311    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c646aa7-6ffe-4a92-84cc-35139428cccd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8c646aa7-6ffe-4a92-84cc-35139428cccd" (UID: "8c646aa7-6ffe-4a92-84cc-35139428cccd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:00:42.001430 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:42.001413    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c646aa7-6ffe-4a92-84cc-35139428cccd-model-cache" (OuterVolumeSpecName: "model-cache") pod "8c646aa7-6ffe-4a92-84cc-35139428cccd" (UID: "8c646aa7-6ffe-4a92-84cc-35139428cccd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:00:42.003186 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:42.003161    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c646aa7-6ffe-4a92-84cc-35139428cccd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8c646aa7-6ffe-4a92-84cc-35139428cccd" (UID: "8c646aa7-6ffe-4a92-84cc-35139428cccd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:00:42.003278 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:42.003242    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c646aa7-6ffe-4a92-84cc-35139428cccd-kube-api-access-6lwrf" (OuterVolumeSpecName: "kube-api-access-6lwrf") pod "8c646aa7-6ffe-4a92-84cc-35139428cccd" (UID: "8c646aa7-6ffe-4a92-84cc-35139428cccd"). InnerVolumeSpecName "kube-api-access-6lwrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:00:42.003937 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:42.003922    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c646aa7-6ffe-4a92-84cc-35139428cccd-dshm" (OuterVolumeSpecName: "dshm") pod "8c646aa7-6ffe-4a92-84cc-35139428cccd" (UID: "8c646aa7-6ffe-4a92-84cc-35139428cccd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:00:42.026153 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:42.026130    2577 scope.go:117] "RemoveContainer" containerID="85c4ca5bea901a9ee1c0a766c354c84601c0e186889e82d1a12680ae5983efe6"
Sep 16 13:00:42.026484 ip-10-0-133-190 kubenswrapper[2577]: E0916 13:00:42.026463    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85c4ca5bea901a9ee1c0a766c354c84601c0e186889e82d1a12680ae5983efe6\": container with ID starting with 85c4ca5bea901a9ee1c0a766c354c84601c0e186889e82d1a12680ae5983efe6 not found: ID does not exist" containerID="85c4ca5bea901a9ee1c0a766c354c84601c0e186889e82d1a12680ae5983efe6"
Sep 16 13:00:42.026552 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:42.026492    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85c4ca5bea901a9ee1c0a766c354c84601c0e186889e82d1a12680ae5983efe6"} err="failed to get container status \"85c4ca5bea901a9ee1c0a766c354c84601c0e186889e82d1a12680ae5983efe6\": rpc error: code = NotFound desc = could not find container \"85c4ca5bea901a9ee1c0a766c354c84601c0e186889e82d1a12680ae5983efe6\": container with ID starting with 85c4ca5bea901a9ee1c0a766c354c84601c0e186889e82d1a12680ae5983efe6 not found: ID does not exist"
Sep 16 13:00:42.102312 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:42.102248    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c646aa7-6ffe-4a92-84cc-35139428cccd-dshm\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:00:42.102312 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:42.102272    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c646aa7-6ffe-4a92-84cc-35139428cccd-tls-certs\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:00:42.102312 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:42.102281    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c646aa7-6ffe-4a92-84cc-35139428cccd-model-cache\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:00:42.102312 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:42.102289    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c646aa7-6ffe-4a92-84cc-35139428cccd-tmp-dir\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:00:42.102312 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:42.102299    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lwrf\" (UniqueName: \"kubernetes.io/projected/8c646aa7-6ffe-4a92-84cc-35139428cccd-kube-api-access-6lwrf\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:00:42.338366 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:42.338328    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-7087d2/explicit-model-7087d2-kserve-64fd59598f-55jpb"]
Sep 16 13:00:42.359142 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:42.359067    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-7087d2/explicit-model-7087d2-kserve-64fd59598f-55jpb"]
Sep 16 13:00:43.551982 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:43.551947    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-586bab/test-model-586bab-kserve-55d4dd4689-k5q5k"]
Sep 16 13:00:44.055181 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:44.055145    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c646aa7-6ffe-4a92-84cc-35139428cccd" path="/var/lib/kubelet/pods/8c646aa7-6ffe-4a92-84cc-35139428cccd/volumes"
Sep 16 13:00:45.504892 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.504859    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"]
Sep 16 13:00:45.505379 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.505363    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c05b6985-688b-4730-ab03-bd11f2b42a6c" containerName="revoke-keys"
Sep 16 13:00:45.505379 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.505380    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c05b6985-688b-4730-ab03-bd11f2b42a6c" containerName="revoke-keys"
Sep 16 13:00:45.505450 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.505397    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c88bd1ea-698b-480b-9a3f-674354738593" containerName="payload-processing"
Sep 16 13:00:45.505450 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.505404    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c88bd1ea-698b-480b-9a3f-674354738593" containerName="payload-processing"
Sep 16 13:00:45.505450 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.505422    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c646aa7-6ffe-4a92-84cc-35139428cccd" containerName="main"
Sep 16 13:00:45.505450 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.505428    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c646aa7-6ffe-4a92-84cc-35139428cccd" containerName="main"
Sep 16 13:00:45.505595 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.505489    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c646aa7-6ffe-4a92-84cc-35139428cccd" containerName="main"
Sep 16 13:00:45.505595 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.505498    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c88bd1ea-698b-480b-9a3f-674354738593" containerName="payload-processing"
Sep 16 13:00:45.505595 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.505505    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c05b6985-688b-4730-ab03-bd11f2b42a6c" containerName="revoke-keys"
Sep 16 13:00:45.508795 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.508777    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"
Sep 16 13:00:45.512216 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.512198    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-b159da-openshift-default-dockercfg-jhp5g\""
Sep 16 13:00:45.519931 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.519906    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"]
Sep 16 13:00:45.535735 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.535711    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/898bb80d-221e-419f-a1c7-d9a28ced4e99-workload-socket\") pod \"e2e-noten-b159da-openshift-default-788696c96-2mmrv\" (UID: \"898bb80d-221e-419f-a1c7-d9a28ced4e99\") " pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"
Sep 16 13:00:45.535860 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.535761    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/898bb80d-221e-419f-a1c7-d9a28ced4e99-istiod-ca-cert\") pod \"e2e-noten-b159da-openshift-default-788696c96-2mmrv\" (UID: \"898bb80d-221e-419f-a1c7-d9a28ced4e99\") " pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"
Sep 16 13:00:45.535860 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.535801    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9xjf\" (UniqueName: \"kubernetes.io/projected/898bb80d-221e-419f-a1c7-d9a28ced4e99-kube-api-access-w9xjf\") pod \"e2e-noten-b159da-openshift-default-788696c96-2mmrv\" (UID: \"898bb80d-221e-419f-a1c7-d9a28ced4e99\") " pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"
Sep 16 13:00:45.535937 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.535867    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/898bb80d-221e-419f-a1c7-d9a28ced4e99-istio-data\") pod \"e2e-noten-b159da-openshift-default-788696c96-2mmrv\" (UID: \"898bb80d-221e-419f-a1c7-d9a28ced4e99\") " pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"
Sep 16 13:00:45.535973 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.535934    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/898bb80d-221e-419f-a1c7-d9a28ced4e99-istio-token\") pod \"e2e-noten-b159da-openshift-default-788696c96-2mmrv\" (UID: \"898bb80d-221e-419f-a1c7-d9a28ced4e99\") " pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"
Sep 16 13:00:45.536008 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.535973    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/898bb80d-221e-419f-a1c7-d9a28ced4e99-istio-podinfo\") pod \"e2e-noten-b159da-openshift-default-788696c96-2mmrv\" (UID: \"898bb80d-221e-419f-a1c7-d9a28ced4e99\") " pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"
Sep 16 13:00:45.536075 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.536062    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/898bb80d-221e-419f-a1c7-d9a28ced4e99-istio-envoy\") pod \"e2e-noten-b159da-openshift-default-788696c96-2mmrv\" (UID: \"898bb80d-221e-419f-a1c7-d9a28ced4e99\") " pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"
Sep 16 13:00:45.536117 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.536094    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/898bb80d-221e-419f-a1c7-d9a28ced4e99-workload-certs\") pod \"e2e-noten-b159da-openshift-default-788696c96-2mmrv\" (UID: \"898bb80d-221e-419f-a1c7-d9a28ced4e99\") " pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"
Sep 16 13:00:45.536164 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.536148    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/898bb80d-221e-419f-a1c7-d9a28ced4e99-credential-socket\") pod \"e2e-noten-b159da-openshift-default-788696c96-2mmrv\" (UID: \"898bb80d-221e-419f-a1c7-d9a28ced4e99\") " pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"
Sep 16 13:00:45.636979 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.636946    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/898bb80d-221e-419f-a1c7-d9a28ced4e99-credential-socket\") pod \"e2e-noten-b159da-openshift-default-788696c96-2mmrv\" (UID: \"898bb80d-221e-419f-a1c7-d9a28ced4e99\") " pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"
Sep 16 13:00:45.636979 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.636979    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/898bb80d-221e-419f-a1c7-d9a28ced4e99-workload-socket\") pod \"e2e-noten-b159da-openshift-default-788696c96-2mmrv\" (UID: \"898bb80d-221e-419f-a1c7-d9a28ced4e99\") " pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"
Sep 16 13:00:45.637227 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.637019    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/898bb80d-221e-419f-a1c7-d9a28ced4e99-istiod-ca-cert\") pod \"e2e-noten-b159da-openshift-default-788696c96-2mmrv\" (UID: \"898bb80d-221e-419f-a1c7-d9a28ced4e99\") " pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"
Sep 16 13:00:45.637227 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.637093    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9xjf\" (UniqueName: \"kubernetes.io/projected/898bb80d-221e-419f-a1c7-d9a28ced4e99-kube-api-access-w9xjf\") pod \"e2e-noten-b159da-openshift-default-788696c96-2mmrv\" (UID: \"898bb80d-221e-419f-a1c7-d9a28ced4e99\") " pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"
Sep 16 13:00:45.637355 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.637219    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/898bb80d-221e-419f-a1c7-d9a28ced4e99-istio-data\") pod \"e2e-noten-b159da-openshift-default-788696c96-2mmrv\" (UID: \"898bb80d-221e-419f-a1c7-d9a28ced4e99\") " pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"
Sep 16 13:00:45.637355 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.637257    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/898bb80d-221e-419f-a1c7-d9a28ced4e99-istio-token\") pod \"e2e-noten-b159da-openshift-default-788696c96-2mmrv\" (UID: \"898bb80d-221e-419f-a1c7-d9a28ced4e99\") " pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"
Sep 16 13:00:45.637355 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.637284    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/898bb80d-221e-419f-a1c7-d9a28ced4e99-istio-podinfo\") pod \"e2e-noten-b159da-openshift-default-788696c96-2mmrv\" (UID: \"898bb80d-221e-419f-a1c7-d9a28ced4e99\") " pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"
Sep 16 13:00:45.637355 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.637333    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/898bb80d-221e-419f-a1c7-d9a28ced4e99-istio-envoy\") pod \"e2e-noten-b159da-openshift-default-788696c96-2mmrv\" (UID: \"898bb80d-221e-419f-a1c7-d9a28ced4e99\") " pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"
Sep 16 13:00:45.637784 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.637369    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/898bb80d-221e-419f-a1c7-d9a28ced4e99-workload-certs\") pod \"e2e-noten-b159da-openshift-default-788696c96-2mmrv\" (UID: \"898bb80d-221e-419f-a1c7-d9a28ced4e99\") " pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"
Sep 16 13:00:45.637784 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.637438    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/898bb80d-221e-419f-a1c7-d9a28ced4e99-workload-socket\") pod \"e2e-noten-b159da-openshift-default-788696c96-2mmrv\" (UID: \"898bb80d-221e-419f-a1c7-d9a28ced4e99\") " pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"
Sep 16 13:00:45.637784 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.637450    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/898bb80d-221e-419f-a1c7-d9a28ced4e99-credential-socket\") pod \"e2e-noten-b159da-openshift-default-788696c96-2mmrv\" (UID: \"898bb80d-221e-419f-a1c7-d9a28ced4e99\") " pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"
Sep 16 13:00:45.637784 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.637617    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/898bb80d-221e-419f-a1c7-d9a28ced4e99-istio-data\") pod \"e2e-noten-b159da-openshift-default-788696c96-2mmrv\" (UID: \"898bb80d-221e-419f-a1c7-d9a28ced4e99\") " pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"
Sep 16 13:00:45.637784 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.637727    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/898bb80d-221e-419f-a1c7-d9a28ced4e99-workload-certs\") pod \"e2e-noten-b159da-openshift-default-788696c96-2mmrv\" (UID: \"898bb80d-221e-419f-a1c7-d9a28ced4e99\") " pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"
Sep 16 13:00:45.638014 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.637808    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/898bb80d-221e-419f-a1c7-d9a28ced4e99-istiod-ca-cert\") pod \"e2e-noten-b159da-openshift-default-788696c96-2mmrv\" (UID: \"898bb80d-221e-419f-a1c7-d9a28ced4e99\") " pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"
Sep 16 13:00:45.639654 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.639630    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/898bb80d-221e-419f-a1c7-d9a28ced4e99-istio-envoy\") pod \"e2e-noten-b159da-openshift-default-788696c96-2mmrv\" (UID: \"898bb80d-221e-419f-a1c7-d9a28ced4e99\") " pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"
Sep 16 13:00:45.640069 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.640047    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/898bb80d-221e-419f-a1c7-d9a28ced4e99-istio-podinfo\") pod \"e2e-noten-b159da-openshift-default-788696c96-2mmrv\" (UID: \"898bb80d-221e-419f-a1c7-d9a28ced4e99\") " pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"
Sep 16 13:00:45.650086 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.650065    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9xjf\" (UniqueName: \"kubernetes.io/projected/898bb80d-221e-419f-a1c7-d9a28ced4e99-kube-api-access-w9xjf\") pod \"e2e-noten-b159da-openshift-default-788696c96-2mmrv\" (UID: \"898bb80d-221e-419f-a1c7-d9a28ced4e99\") " pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"
Sep 16 13:00:45.650698 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.650682    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/898bb80d-221e-419f-a1c7-d9a28ced4e99-istio-token\") pod \"e2e-noten-b159da-openshift-default-788696c96-2mmrv\" (UID: \"898bb80d-221e-419f-a1c7-d9a28ced4e99\") " pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"
Sep 16 13:00:45.820868 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:45.820784    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"
Sep 16 13:00:46.114531 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:46.114502    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"]
Sep 16 13:00:46.117117 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:00:46.117092    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod898bb80d_221e_419f_a1c7_d9a28ced4e99.slice/crio-0256468f0cd585edebd5a1a666d35c3b5328c0e0a0adccb3959384d9811c4fd5 WatchSource:0}: Error finding container 0256468f0cd585edebd5a1a666d35c3b5328c0e0a0adccb3959384d9811c4fd5: Status 404 returned error can't find the container with id 0256468f0cd585edebd5a1a666d35c3b5328c0e0a0adccb3959384d9811c4fd5
Sep 16 13:00:46.119383 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:46.119350    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 13:00:46.119477 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:46.119426    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 13:00:46.119477 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:46.119468    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 13:00:47.035163 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:47.035128    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv" event={"ID":"898bb80d-221e-419f-a1c7-d9a28ced4e99","Type":"ContainerStarted","Data":"b3a4cf81d668ae35fd2669f3a0cdddfe5751fe14bf633229c656b3ce9b58ec10"}
Sep 16 13:00:47.035533 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:47.035180    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv" event={"ID":"898bb80d-221e-419f-a1c7-d9a28ced4e99","Type":"ContainerStarted","Data":"0256468f0cd585edebd5a1a666d35c3b5328c0e0a0adccb3959384d9811c4fd5"}
Sep 16 13:00:47.066636 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:47.066590    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv" podStartSLOduration=2.06657026 podStartE2EDuration="2.06657026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:00:47.064923791 +0000 UTC m=+1353.512156433" watchObservedRunningTime="2026-09-16 13:00:47.06657026 +0000 UTC m=+1353.513802902"
Sep 16 13:00:47.821707 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:47.821677    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"
Sep 16 13:00:47.827120 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:47.827096    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"
Sep 16 13:00:48.041625 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:48.041589    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"
Sep 16 13:00:48.042595 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:48.042576    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"
Sep 16 13:00:58.571446 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:58.571380    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-586bab/test-model-586bab-kserve-55d4dd4689-k5q5k" podUID="7faf6052-c278-496d-9ebc-95d90b2d193d" containerName="main" containerID="cri-o://fd5a8fbadda9d11d69d5969290fe3939b29b44d1da9e31c26c3224a190d0f557" gracePeriod=45
Sep 16 13:00:58.834017 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:58.833952    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-586bab/test-model-586bab-kserve-55d4dd4689-k5q5k"
Sep 16 13:00:58.977886 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:58.977857    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7faf6052-c278-496d-9ebc-95d90b2d193d-dshm\") pod \"7faf6052-c278-496d-9ebc-95d90b2d193d\" (UID: \"7faf6052-c278-496d-9ebc-95d90b2d193d\") "
Sep 16 13:00:58.978122 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:58.977903    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7faf6052-c278-496d-9ebc-95d90b2d193d-model-cache\") pod \"7faf6052-c278-496d-9ebc-95d90b2d193d\" (UID: \"7faf6052-c278-496d-9ebc-95d90b2d193d\") "
Sep 16 13:00:58.978122 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:58.977954    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7faf6052-c278-496d-9ebc-95d90b2d193d-tls-certs\") pod \"7faf6052-c278-496d-9ebc-95d90b2d193d\" (UID: \"7faf6052-c278-496d-9ebc-95d90b2d193d\") "
Sep 16 13:00:58.978122 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:58.977974    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7faf6052-c278-496d-9ebc-95d90b2d193d-tmp-dir\") pod \"7faf6052-c278-496d-9ebc-95d90b2d193d\" (UID: \"7faf6052-c278-496d-9ebc-95d90b2d193d\") "
Sep 16 13:00:58.978122 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:58.978082    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5x5q\" (UniqueName: \"kubernetes.io/projected/7faf6052-c278-496d-9ebc-95d90b2d193d-kube-api-access-g5x5q\") pod \"7faf6052-c278-496d-9ebc-95d90b2d193d\" (UID: \"7faf6052-c278-496d-9ebc-95d90b2d193d\") "
Sep 16 13:00:58.978122 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:58.978115    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7faf6052-c278-496d-9ebc-95d90b2d193d-home\") pod \"7faf6052-c278-496d-9ebc-95d90b2d193d\" (UID: \"7faf6052-c278-496d-9ebc-95d90b2d193d\") "
Sep 16 13:00:58.978422 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:58.978277    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7faf6052-c278-496d-9ebc-95d90b2d193d-model-cache" (OuterVolumeSpecName: "model-cache") pod "7faf6052-c278-496d-9ebc-95d90b2d193d" (UID: "7faf6052-c278-496d-9ebc-95d90b2d193d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:00:58.978422 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:58.978296    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7faf6052-c278-496d-9ebc-95d90b2d193d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7faf6052-c278-496d-9ebc-95d90b2d193d" (UID: "7faf6052-c278-496d-9ebc-95d90b2d193d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:00:58.978497 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:58.978418    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7faf6052-c278-496d-9ebc-95d90b2d193d-home" (OuterVolumeSpecName: "home") pod "7faf6052-c278-496d-9ebc-95d90b2d193d" (UID: "7faf6052-c278-496d-9ebc-95d90b2d193d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:00:58.978584 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:58.978567    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7faf6052-c278-496d-9ebc-95d90b2d193d-tmp-dir\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:00:58.978623 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:58.978587    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7faf6052-c278-496d-9ebc-95d90b2d193d-home\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:00:58.978623 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:58.978595    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7faf6052-c278-496d-9ebc-95d90b2d193d-model-cache\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:00:58.980349 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:58.980325    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7faf6052-c278-496d-9ebc-95d90b2d193d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7faf6052-c278-496d-9ebc-95d90b2d193d" (UID: "7faf6052-c278-496d-9ebc-95d90b2d193d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:00:58.980777 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:58.980761    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7faf6052-c278-496d-9ebc-95d90b2d193d-dshm" (OuterVolumeSpecName: "dshm") pod "7faf6052-c278-496d-9ebc-95d90b2d193d" (UID: "7faf6052-c278-496d-9ebc-95d90b2d193d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:00:58.980843 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:58.980785    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7faf6052-c278-496d-9ebc-95d90b2d193d-kube-api-access-g5x5q" (OuterVolumeSpecName: "kube-api-access-g5x5q") pod "7faf6052-c278-496d-9ebc-95d90b2d193d" (UID: "7faf6052-c278-496d-9ebc-95d90b2d193d"). InnerVolumeSpecName "kube-api-access-g5x5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:00:59.079620 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:59.079590    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7faf6052-c278-496d-9ebc-95d90b2d193d-dshm\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:00:59.079620 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:59.079617    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7faf6052-c278-496d-9ebc-95d90b2d193d-tls-certs\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:00:59.079620 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:59.079628    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5x5q\" (UniqueName: \"kubernetes.io/projected/7faf6052-c278-496d-9ebc-95d90b2d193d-kube-api-access-g5x5q\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:00:59.110365 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:59.110280    2577 generic.go:356] "Generic (PLEG): container finished" podID="7faf6052-c278-496d-9ebc-95d90b2d193d" containerID="fd5a8fbadda9d11d69d5969290fe3939b29b44d1da9e31c26c3224a190d0f557" exitCode=0
Sep 16 13:00:59.110365 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:59.110335    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-586bab/test-model-586bab-kserve-55d4dd4689-k5q5k"
Sep 16 13:00:59.110592 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:59.110363    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-586bab/test-model-586bab-kserve-55d4dd4689-k5q5k" event={"ID":"7faf6052-c278-496d-9ebc-95d90b2d193d","Type":"ContainerDied","Data":"fd5a8fbadda9d11d69d5969290fe3939b29b44d1da9e31c26c3224a190d0f557"}
Sep 16 13:00:59.110592 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:59.110412    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-586bab/test-model-586bab-kserve-55d4dd4689-k5q5k" event={"ID":"7faf6052-c278-496d-9ebc-95d90b2d193d","Type":"ContainerDied","Data":"27a9a95d705e880a96d8c14dac10a4ec6f00a06a6d9ae04113669bda79bc725b"}
Sep 16 13:00:59.110592 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:59.110435    2577 scope.go:117] "RemoveContainer" containerID="fd5a8fbadda9d11d69d5969290fe3939b29b44d1da9e31c26c3224a190d0f557"
Sep 16 13:00:59.121061 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:59.121041    2577 scope.go:117] "RemoveContainer" containerID="fd5a8fbadda9d11d69d5969290fe3939b29b44d1da9e31c26c3224a190d0f557"
Sep 16 13:00:59.121388 ip-10-0-133-190 kubenswrapper[2577]: E0916 13:00:59.121367    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd5a8fbadda9d11d69d5969290fe3939b29b44d1da9e31c26c3224a190d0f557\": container with ID starting with fd5a8fbadda9d11d69d5969290fe3939b29b44d1da9e31c26c3224a190d0f557 not found: ID does not exist" containerID="fd5a8fbadda9d11d69d5969290fe3939b29b44d1da9e31c26c3224a190d0f557"
Sep 16 13:00:59.121454 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:59.121401    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd5a8fbadda9d11d69d5969290fe3939b29b44d1da9e31c26c3224a190d0f557"} err="failed to get container status \"fd5a8fbadda9d11d69d5969290fe3939b29b44d1da9e31c26c3224a190d0f557\": rpc error: code = NotFound desc = could not find container \"fd5a8fbadda9d11d69d5969290fe3939b29b44d1da9e31c26c3224a190d0f557\": container with ID starting with fd5a8fbadda9d11d69d5969290fe3939b29b44d1da9e31c26c3224a190d0f557 not found: ID does not exist"
Sep 16 13:00:59.136232 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:59.136209    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-586bab/test-model-586bab-kserve-55d4dd4689-k5q5k"]
Sep 16 13:00:59.141724 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:00:59.141699    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-586bab/test-model-586bab-kserve-55d4dd4689-k5q5k"]
Sep 16 13:01:00.055488 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:00.055453    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7faf6052-c278-496d-9ebc-95d90b2d193d" path="/var/lib/kubelet/pods/7faf6052-c278-496d-9ebc-95d90b2d193d/volumes"
Sep 16 13:01:01.282726 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:01.282693    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"]
Sep 16 13:01:01.283168 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:01.283155    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7faf6052-c278-496d-9ebc-95d90b2d193d" containerName="main"
Sep 16 13:01:01.283168 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:01.283169    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7faf6052-c278-496d-9ebc-95d90b2d193d" containerName="main"
Sep 16 13:01:01.283256 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:01.283239    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7faf6052-c278-496d-9ebc-95d90b2d193d" containerName="main"
Sep 16 13:01:01.287921 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:01.287896    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"
Sep 16 13:01:01.291435 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:01.291411    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-e5926f-openshift-default-dockercfg-czzd5\""
Sep 16 13:01:01.313610 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:01.313585    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"]
Sep 16 13:01:01.406684 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:01.406643    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05103a7a-76fc-4417-a62a-2d1439edc444-credential-socket\") pod \"e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn\" (UID: \"05103a7a-76fc-4417-a62a-2d1439edc444\") " pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"
Sep 16 13:01:01.406684 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:01.406684    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05103a7a-76fc-4417-a62a-2d1439edc444-workload-certs\") pod \"e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn\" (UID: \"05103a7a-76fc-4417-a62a-2d1439edc444\") " pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"
Sep 16 13:01:01.406895 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:01.406719    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7ffw\" (UniqueName: \"kubernetes.io/projected/05103a7a-76fc-4417-a62a-2d1439edc444-kube-api-access-r7ffw\") pod \"e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn\" (UID: \"05103a7a-76fc-4417-a62a-2d1439edc444\") " pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"
Sep 16 13:01:01.406895 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:01.406794    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05103a7a-76fc-4417-a62a-2d1439edc444-istio-token\") pod \"e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn\" (UID: \"05103a7a-76fc-4417-a62a-2d1439edc444\") " pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"
Sep 16 13:01:01.406895 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:01.406831    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05103a7a-76fc-4417-a62a-2d1439edc444-istio-data\") pod \"e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn\" (UID: \"05103a7a-76fc-4417-a62a-2d1439edc444\") " pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"
Sep 16 13:01:01.406895 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:01.406859    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05103a7a-76fc-4417-a62a-2d1439edc444-istio-podinfo\") pod \"e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn\" (UID: \"05103a7a-76fc-4417-a62a-2d1439edc444\") " pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"
Sep 16 13:01:01.406895 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:01.406880    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05103a7a-76fc-4417-a62a-2d1439edc444-workload-socket\") pod \"e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn\" (UID: \"05103a7a-76fc-4417-a62a-2d1439edc444\") " pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"
Sep 16 13:01:01.407174 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:01.406927    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05103a7a-76fc-4417-a62a-2d1439edc444-istio-envoy\") pod \"e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn\" (UID: \"05103a7a-76fc-4417-a62a-2d1439edc444\") " pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"
Sep 16 13:01:01.407174 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:01.407066    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05103a7a-76fc-4417-a62a-2d1439edc444-istiod-ca-cert\") pod \"e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn\" (UID: \"05103a7a-76fc-4417-a62a-2d1439edc444\") " pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"
Sep 16 13:01:01.508261 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:01.508222    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05103a7a-76fc-4417-a62a-2d1439edc444-workload-certs\") pod \"e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn\" (UID: \"05103a7a-76fc-4417-a62a-2d1439edc444\") " pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"
Sep 16 13:01:01.508434 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:01.508282    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7ffw\" (UniqueName: \"kubernetes.io/projected/05103a7a-76fc-4417-a62a-2d1439edc444-kube-api-access-r7ffw\") pod \"e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn\" (UID: \"05103a7a-76fc-4417-a62a-2d1439edc444\") " pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"
Sep 16 13:01:01.508434 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:01.508337    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05103a7a-76fc-4417-a62a-2d1439edc444-istio-token\") pod \"e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn\" (UID: \"05103a7a-76fc-4417-a62a-2d1439edc444\") " pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"
Sep 16 13:01:01.508434 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:01.508368    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05103a7a-76fc-4417-a62a-2d1439edc444-istio-data\") pod \"e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn\" (UID: \"05103a7a-76fc-4417-a62a-2d1439edc444\") " pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"
Sep 16 13:01:01.508434 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:01.508396    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05103a7a-76fc-4417-a62a-2d1439edc444-istio-podinfo\") pod \"e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn\" (UID: \"05103a7a-76fc-4417-a62a-2d1439edc444\") " pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"
Sep 16 13:01:01.508434 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:01.508422    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05103a7a-76fc-4417-a62a-2d1439edc444-workload-socket\") pod \"e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn\" (UID: \"05103a7a-76fc-4417-a62a-2d1439edc444\") " pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"
Sep 16 13:01:01.508708 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:01.508502    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05103a7a-76fc-4417-a62a-2d1439edc444-istio-envoy\") pod \"e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn\" (UID: \"05103a7a-76fc-4417-a62a-2d1439edc444\") " pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"
Sep 16 13:01:01.508708 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:01.508576    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05103a7a-76fc-4417-a62a-2d1439edc444-istiod-ca-cert\") pod \"e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn\" (UID: \"05103a7a-76fc-4417-a62a-2d1439edc444\") " pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"
Sep 16 13:01:01.508708 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:01.508648    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05103a7a-76fc-4417-a62a-2d1439edc444-credential-socket\") pod \"e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn\" (UID: \"05103a7a-76fc-4417-a62a-2d1439edc444\") " pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"
Sep 16 13:01:01.508708 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:01.508684    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05103a7a-76fc-4417-a62a-2d1439edc444-workload-certs\") pod \"e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn\" (UID: \"05103a7a-76fc-4417-a62a-2d1439edc444\") " pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"
Sep 16 13:01:01.508919 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:01.508806    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05103a7a-76fc-4417-a62a-2d1439edc444-workload-socket\") pod \"e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn\" (UID: \"05103a7a-76fc-4417-a62a-2d1439edc444\") " pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"
Sep 16 13:01:01.508919 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:01.508883    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05103a7a-76fc-4417-a62a-2d1439edc444-credential-socket\") pod \"e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn\" (UID: \"05103a7a-76fc-4417-a62a-2d1439edc444\") " pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"
Sep 16 13:01:01.509047 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:01.509007    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05103a7a-76fc-4417-a62a-2d1439edc444-istio-data\") pod \"e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn\" (UID: \"05103a7a-76fc-4417-a62a-2d1439edc444\") " pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"
Sep 16 13:01:01.509261 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:01.509245    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05103a7a-76fc-4417-a62a-2d1439edc444-istiod-ca-cert\") pod \"e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn\" (UID: \"05103a7a-76fc-4417-a62a-2d1439edc444\") " pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"
Sep 16 13:01:01.510867 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:01.510848    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05103a7a-76fc-4417-a62a-2d1439edc444-istio-envoy\") pod \"e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn\" (UID: \"05103a7a-76fc-4417-a62a-2d1439edc444\") " pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"
Sep 16 13:01:01.511212 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:01.511191    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05103a7a-76fc-4417-a62a-2d1439edc444-istio-podinfo\") pod \"e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn\" (UID: \"05103a7a-76fc-4417-a62a-2d1439edc444\") " pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"
Sep 16 13:01:01.518382 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:01.518362    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05103a7a-76fc-4417-a62a-2d1439edc444-istio-token\") pod \"e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn\" (UID: \"05103a7a-76fc-4417-a62a-2d1439edc444\") " pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"
Sep 16 13:01:01.518719 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:01.518699    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7ffw\" (UniqueName: \"kubernetes.io/projected/05103a7a-76fc-4417-a62a-2d1439edc444-kube-api-access-r7ffw\") pod \"e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn\" (UID: \"05103a7a-76fc-4417-a62a-2d1439edc444\") " pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"
Sep 16 13:01:01.600045 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:01.599951    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"
Sep 16 13:01:01.892467 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:01.892435    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"]
Sep 16 13:01:01.892587 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:01:01.892510    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05103a7a_76fc_4417_a62a_2d1439edc444.slice/crio-4f784af204d06c73f46387627c8f80011f2ee21c567be3d930119fbb17f9f87c WatchSource:0}: Error finding container 4f784af204d06c73f46387627c8f80011f2ee21c567be3d930119fbb17f9f87c: Status 404 returned error can't find the container with id 4f784af204d06c73f46387627c8f80011f2ee21c567be3d930119fbb17f9f87c
Sep 16 13:01:01.895180 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:01.895144    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 13:01:01.895256 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:01.895227    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 13:01:01.895296 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:01.895275    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 13:01:02.135976 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:02.135944    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn" event={"ID":"05103a7a-76fc-4417-a62a-2d1439edc444","Type":"ContainerStarted","Data":"4597b6bb5db08ddf331310f64c8e1f1529b730ffd6466a285d45ef1b97e58640"}
Sep 16 13:01:02.136207 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:02.136004    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn" event={"ID":"05103a7a-76fc-4417-a62a-2d1439edc444","Type":"ContainerStarted","Data":"4f784af204d06c73f46387627c8f80011f2ee21c567be3d930119fbb17f9f87c"}
Sep 16 13:01:02.164159 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:02.164051    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn" podStartSLOduration=1.164007818 podStartE2EDuration="1.164007818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:01:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:01:02.159189741 +0000 UTC m=+1368.606422383" watchObservedRunningTime="2026-09-16 13:01:02.164007818 +0000 UTC m=+1368.611240461"
Sep 16 13:01:02.600293 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:02.600256    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"
Sep 16 13:01:02.605512 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:02.605489    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"
Sep 16 13:01:03.142489 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:03.142455    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"
Sep 16 13:01:03.143525 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:03.143505    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"
Sep 16 13:01:05.542696 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:05.542654    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-89d6bf/test-model-89d6bf-kserve-844b98bc79-xwdp2"]
Sep 16 13:01:05.707298 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:05.707263    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-586bab-k5b9h"]
Sep 16 13:01:05.711126 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:05.711108    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-586bab-k5b9h"
Sep 16 13:01:05.715971 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:05.715949    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 13:01:05.716143 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:05.715986    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-64zph\""
Sep 16 13:01:05.720504 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:05.720483    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-586bab-k5b9h"]
Sep 16 13:01:05.855902 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:05.855815    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c224a8b-6133-4ed7-bb2d-b512065d6675-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-586bab-k5b9h\" (UID: \"4c224a8b-6133-4ed7-bb2d-b512065d6675\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-586bab-k5b9h"
Sep 16 13:01:05.957039 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:05.956987    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c224a8b-6133-4ed7-bb2d-b512065d6675-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-586bab-k5b9h\" (UID: \"4c224a8b-6133-4ed7-bb2d-b512065d6675\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-586bab-k5b9h"
Sep 16 13:01:05.957627 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:05.957597    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c224a8b-6133-4ed7-bb2d-b512065d6675-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-586bab-k5b9h\" (UID: \"4c224a8b-6133-4ed7-bb2d-b512065d6675\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-586bab-k5b9h"
Sep 16 13:01:06.023983 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:06.023952    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-586bab-k5b9h"
Sep 16 13:01:06.303874 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:06.303842    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-586bab-k5b9h"]
Sep 16 13:01:06.304466 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:01:06.304442    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c224a8b_6133_4ed7_bb2d_b512065d6675.slice/crio-b9ba3b885c557fbbe6ada59c148be0f130a2a4f617a63ef028411f59e1f5d4e9 WatchSource:0}: Error finding container b9ba3b885c557fbbe6ada59c148be0f130a2a4f617a63ef028411f59e1f5d4e9: Status 404 returned error can't find the container with id b9ba3b885c557fbbe6ada59c148be0f130a2a4f617a63ef028411f59e1f5d4e9
Sep 16 13:01:07.166996 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:07.166959    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-586bab-k5b9h" event={"ID":"4c224a8b-6133-4ed7-bb2d-b512065d6675","Type":"ContainerStarted","Data":"beb483b9eb16af52efc711f5bd89b9823d296264c1565cb61190cf931858a6a4"}
Sep 16 13:01:07.167378 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:07.167018    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-586bab-k5b9h" event={"ID":"4c224a8b-6133-4ed7-bb2d-b512065d6675","Type":"ContainerStarted","Data":"b9ba3b885c557fbbe6ada59c148be0f130a2a4f617a63ef028411f59e1f5d4e9"}
Sep 16 13:01:07.188252 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:07.188192    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-586bab-k5b9h" podStartSLOduration=2.188171573 podStartE2EDuration="2.188171573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:01:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:01:07.184469563 +0000 UTC m=+1373.631702205" watchObservedRunningTime="2026-09-16 13:01:07.188171573 +0000 UTC m=+1373.635404216"
Sep 16 13:01:08.174999 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:08.174967    2577 generic.go:356] "Generic (PLEG): container finished" podID="4c224a8b-6133-4ed7-bb2d-b512065d6675" containerID="beb483b9eb16af52efc711f5bd89b9823d296264c1565cb61190cf931858a6a4" exitCode=0
Sep 16 13:01:08.175420 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:08.175061    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-586bab-k5b9h" event={"ID":"4c224a8b-6133-4ed7-bb2d-b512065d6675","Type":"ContainerDied","Data":"beb483b9eb16af52efc711f5bd89b9823d296264c1565cb61190cf931858a6a4"}
Sep 16 13:01:09.317508 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:09.317488    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-586bab-k5b9h"
Sep 16 13:01:09.499750 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:09.499670    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c224a8b-6133-4ed7-bb2d-b512065d6675-maas-api-service-ca\") pod \"4c224a8b-6133-4ed7-bb2d-b512065d6675\" (UID: \"4c224a8b-6133-4ed7-bb2d-b512065d6675\") "
Sep 16 13:01:09.500082 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:09.500022    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c224a8b-6133-4ed7-bb2d-b512065d6675-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4c224a8b-6133-4ed7-bb2d-b512065d6675" (UID: "4c224a8b-6133-4ed7-bb2d-b512065d6675"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:01:09.600966 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:09.600934    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c224a8b-6133-4ed7-bb2d-b512065d6675-maas-api-service-ca\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:01:10.189565 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:10.189538    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-586bab-k5b9h"
Sep 16 13:01:10.189738 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:10.189573    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-586bab-k5b9h" event={"ID":"4c224a8b-6133-4ed7-bb2d-b512065d6675","Type":"ContainerDied","Data":"b9ba3b885c557fbbe6ada59c148be0f130a2a4f617a63ef028411f59e1f5d4e9"}
Sep 16 13:01:10.189738 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:10.189629    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9ba3b885c557fbbe6ada59c148be0f130a2a4f617a63ef028411f59e1f5d4e9"
Sep 16 13:01:10.954551 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:10.954517    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5926f-68b6695bbb-cdq2j"]
Sep 16 13:01:10.954985 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:10.954972    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c224a8b-6133-4ed7-bb2d-b512065d6675" containerName="revoke-keys"
Sep 16 13:01:10.955065 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:10.954986    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c224a8b-6133-4ed7-bb2d-b512065d6675" containerName="revoke-keys"
Sep 16 13:01:10.955108 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:10.955093    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c224a8b-6133-4ed7-bb2d-b512065d6675" containerName="revoke-keys"
Sep 16 13:01:10.958285 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:10.958269    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5926f-68b6695bbb-cdq2j"
Sep 16 13:01:10.965808 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:10.965786    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-e5926f\""
Sep 16 13:01:10.995800 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:10.995767    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5926f-68b6695bbb-cdq2j"]
Sep 16 13:01:11.039639 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:11.039608    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e5926f-7bcd585465-ms5wx"]
Sep 16 13:01:11.043343 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:11.043325    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e5926f-7bcd585465-ms5wx"
Sep 16 13:01:11.046316 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:11.046298    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-e5926f-dockercfg-lxw9z\""
Sep 16 13:01:11.079714 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:11.079678    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e5926f-7bcd585465-ms5wx"]
Sep 16 13:01:11.116073 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:11.116007    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxn6q\" (UniqueName: \"kubernetes.io/projected/01d14db3-8e1d-4462-a9a6-f92962543f90-kube-api-access-nxn6q\") pod \"payload-pre-processing-e2e-shared-b-w2-e5926f-68b6695bbb-cdq2j\" (UID: \"01d14db3-8e1d-4462-a9a6-f92962543f90\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5926f-68b6695bbb-cdq2j"
Sep 16 13:01:11.116217 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:11.116181    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d14db3-8e1d-4462-a9a6-f92962543f90-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-e5926f-68b6695bbb-cdq2j\" (UID: \"01d14db3-8e1d-4462-a9a6-f92962543f90\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5926f-68b6695bbb-cdq2j"
Sep 16 13:01:11.217415 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:11.217337    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d14db3-8e1d-4462-a9a6-f92962543f90-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-e5926f-68b6695bbb-cdq2j\" (UID: \"01d14db3-8e1d-4462-a9a6-f92962543f90\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5926f-68b6695bbb-cdq2j"
Sep 16 13:01:11.217415 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:11.217407    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7mjl\" (UniqueName: \"kubernetes.io/projected/01a3c352-0789-42df-8d81-3242438a84c2-kube-api-access-l7mjl\") pod \"payload-processing-e2e-shared-b-w2-e5926f-7bcd585465-ms5wx\" (UID: \"01a3c352-0789-42df-8d81-3242438a84c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e5926f-7bcd585465-ms5wx"
Sep 16 13:01:11.217599 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:11.217501    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01a3c352-0789-42df-8d81-3242438a84c2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-e5926f-7bcd585465-ms5wx\" (UID: \"01a3c352-0789-42df-8d81-3242438a84c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e5926f-7bcd585465-ms5wx"
Sep 16 13:01:11.217599 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:11.217554    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxn6q\" (UniqueName: \"kubernetes.io/projected/01d14db3-8e1d-4462-a9a6-f92962543f90-kube-api-access-nxn6q\") pod \"payload-pre-processing-e2e-shared-b-w2-e5926f-68b6695bbb-cdq2j\" (UID: \"01d14db3-8e1d-4462-a9a6-f92962543f90\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5926f-68b6695bbb-cdq2j"
Sep 16 13:01:11.218166 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:11.218148    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d14db3-8e1d-4462-a9a6-f92962543f90-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-e5926f-68b6695bbb-cdq2j\" (UID: \"01d14db3-8e1d-4462-a9a6-f92962543f90\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5926f-68b6695bbb-cdq2j"
Sep 16 13:01:11.227419 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:11.227397    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxn6q\" (UniqueName: \"kubernetes.io/projected/01d14db3-8e1d-4462-a9a6-f92962543f90-kube-api-access-nxn6q\") pod \"payload-pre-processing-e2e-shared-b-w2-e5926f-68b6695bbb-cdq2j\" (UID: \"01d14db3-8e1d-4462-a9a6-f92962543f90\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5926f-68b6695bbb-cdq2j"
Sep 16 13:01:11.269150 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:11.269124    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5926f-68b6695bbb-cdq2j"
Sep 16 13:01:11.319082 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:11.318571    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01a3c352-0789-42df-8d81-3242438a84c2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-e5926f-7bcd585465-ms5wx\" (UID: \"01a3c352-0789-42df-8d81-3242438a84c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e5926f-7bcd585465-ms5wx"
Sep 16 13:01:11.319082 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:11.318735    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7mjl\" (UniqueName: \"kubernetes.io/projected/01a3c352-0789-42df-8d81-3242438a84c2-kube-api-access-l7mjl\") pod \"payload-processing-e2e-shared-b-w2-e5926f-7bcd585465-ms5wx\" (UID: \"01a3c352-0789-42df-8d81-3242438a84c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e5926f-7bcd585465-ms5wx"
Sep 16 13:01:11.319382 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:11.319355    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01a3c352-0789-42df-8d81-3242438a84c2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-e5926f-7bcd585465-ms5wx\" (UID: \"01a3c352-0789-42df-8d81-3242438a84c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e5926f-7bcd585465-ms5wx"
Sep 16 13:01:11.330376 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:11.330345    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7mjl\" (UniqueName: \"kubernetes.io/projected/01a3c352-0789-42df-8d81-3242438a84c2-kube-api-access-l7mjl\") pod \"payload-processing-e2e-shared-b-w2-e5926f-7bcd585465-ms5wx\" (UID: \"01a3c352-0789-42df-8d81-3242438a84c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e5926f-7bcd585465-ms5wx"
Sep 16 13:01:11.353564 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:11.353534    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e5926f-7bcd585465-ms5wx"
Sep 16 13:01:11.547736 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:11.547706    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5926f-68b6695bbb-cdq2j"]
Sep 16 13:01:11.548933 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:01:11.548908    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01d14db3_8e1d_4462_a9a6_f92962543f90.slice/crio-740bdcdd1af8e7b0e8cdffa42e16c869e666b459a83f7e25df4b776fb0d094e5 WatchSource:0}: Error finding container 740bdcdd1af8e7b0e8cdffa42e16c869e666b459a83f7e25df4b776fb0d094e5: Status 404 returned error can't find the container with id 740bdcdd1af8e7b0e8cdffa42e16c869e666b459a83f7e25df4b776fb0d094e5
Sep 16 13:01:11.638905 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:11.638875    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e5926f-7bcd585465-ms5wx"]
Sep 16 13:01:11.640360 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:01:11.640332    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a3c352_0789_42df_8d81_3242438a84c2.slice/crio-ce20471bf2620cdae1f8d57de2f60356ce1915df422f35dcf332bdbf9c943101 WatchSource:0}: Error finding container ce20471bf2620cdae1f8d57de2f60356ce1915df422f35dcf332bdbf9c943101: Status 404 returned error can't find the container with id ce20471bf2620cdae1f8d57de2f60356ce1915df422f35dcf332bdbf9c943101
Sep 16 13:01:12.205097 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:12.205054    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e5926f-7bcd585465-ms5wx" event={"ID":"01a3c352-0789-42df-8d81-3242438a84c2","Type":"ContainerStarted","Data":"2d037a63782a56d1c0f3e55a68e666005b5d81620d517323f30c7f4d1eafd27a"}
Sep 16 13:01:12.205549 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:12.205186    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e5926f-7bcd585465-ms5wx"
Sep 16 13:01:12.205549 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:12.205216    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e5926f-7bcd585465-ms5wx" event={"ID":"01a3c352-0789-42df-8d81-3242438a84c2","Type":"ContainerStarted","Data":"ce20471bf2620cdae1f8d57de2f60356ce1915df422f35dcf332bdbf9c943101"}
Sep 16 13:01:12.206841 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:12.206813    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5926f-68b6695bbb-cdq2j" event={"ID":"01d14db3-8e1d-4462-a9a6-f92962543f90","Type":"ContainerStarted","Data":"e50d7008229a715498e2295e2c5bf3f8dc0d13e9f0b2511a5e0b6e37ff94e900"}
Sep 16 13:01:12.206975 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:12.206860    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5926f-68b6695bbb-cdq2j" event={"ID":"01d14db3-8e1d-4462-a9a6-f92962543f90","Type":"ContainerStarted","Data":"740bdcdd1af8e7b0e8cdffa42e16c869e666b459a83f7e25df4b776fb0d094e5"}
Sep 16 13:01:12.206975 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:12.206906    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5926f-68b6695bbb-cdq2j"
Sep 16 13:01:12.231496 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:12.231454    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e5926f-7bcd585465-ms5wx" podStartSLOduration=2.231438197 podStartE2EDuration="2.231438197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:01:12.23041662 +0000 UTC m=+1378.677649266" watchObservedRunningTime="2026-09-16 13:01:12.231438197 +0000 UTC m=+1378.678670842"
Sep 16 13:01:12.256955 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:12.256903    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5926f-68b6695bbb-cdq2j" podStartSLOduration=2.256885403 podStartE2EDuration="2.256885403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:01:12.254216403 +0000 UTC m=+1378.701449046" watchObservedRunningTime="2026-09-16 13:01:12.256885403 +0000 UTC m=+1378.704118047"
Sep 16 13:01:14.696417 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:14.696386    2577 scope.go:117] "RemoveContainer" containerID="f5c41edf4b69f07849bf3e1f4916e2ada130140592ad219e635529a4ed49a25e"
Sep 16 13:01:16.586844 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:16.586811    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-586bab-29826060-962qq"]
Sep 16 13:01:16.590636 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:16.590606    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-586bab-29826060-962qq"]
Sep 16 13:01:16.799016 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:16.798983    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-586bab-k5b9h"]
Sep 16 13:01:16.801400 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:16.801377    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-586bab-k5b9h"]
Sep 16 13:01:18.056069 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:18.056016    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c224a8b-6133-4ed7-bb2d-b512065d6675" path="/var/lib/kubelet/pods/4c224a8b-6133-4ed7-bb2d-b512065d6675/volumes"
Sep 16 13:01:18.056430 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:18.056412    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d778c68c-ac23-476d-97d9-8406b45b31e2" path="/var/lib/kubelet/pods/d778c68c-ac23-476d-97d9-8406b45b31e2/volumes"
Sep 16 13:01:20.562973 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:20.562929    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-89d6bf/test-model-89d6bf-kserve-844b98bc79-xwdp2" podUID="ec8e6f24-cd09-4850-a310-1e52847ee42b" containerName="main" containerID="cri-o://2025fcad32854ceab8f538f1cf6f6137add6be93785656ce45bb83f3a45b056f" gracePeriod=45
Sep 16 13:01:20.835532 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:20.835503    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-89d6bf/test-model-89d6bf-kserve-844b98bc79-xwdp2"
Sep 16 13:01:20.922228 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:20.922193    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec8e6f24-cd09-4850-a310-1e52847ee42b-model-cache\") pod \"ec8e6f24-cd09-4850-a310-1e52847ee42b\" (UID: \"ec8e6f24-cd09-4850-a310-1e52847ee42b\") "
Sep 16 13:01:20.922228 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:20.922230    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgvhc\" (UniqueName: \"kubernetes.io/projected/ec8e6f24-cd09-4850-a310-1e52847ee42b-kube-api-access-xgvhc\") pod \"ec8e6f24-cd09-4850-a310-1e52847ee42b\" (UID: \"ec8e6f24-cd09-4850-a310-1e52847ee42b\") "
Sep 16 13:01:20.922480 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:20.922287    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec8e6f24-cd09-4850-a310-1e52847ee42b-dshm\") pod \"ec8e6f24-cd09-4850-a310-1e52847ee42b\" (UID: \"ec8e6f24-cd09-4850-a310-1e52847ee42b\") "
Sep 16 13:01:20.922480 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:20.922312    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec8e6f24-cd09-4850-a310-1e52847ee42b-tls-certs\") pod \"ec8e6f24-cd09-4850-a310-1e52847ee42b\" (UID: \"ec8e6f24-cd09-4850-a310-1e52847ee42b\") "
Sep 16 13:01:20.922480 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:20.922344    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec8e6f24-cd09-4850-a310-1e52847ee42b-home\") pod \"ec8e6f24-cd09-4850-a310-1e52847ee42b\" (UID: \"ec8e6f24-cd09-4850-a310-1e52847ee42b\") "
Sep 16 13:01:20.922647 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:20.922508    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec8e6f24-cd09-4850-a310-1e52847ee42b-tmp-dir\") pod \"ec8e6f24-cd09-4850-a310-1e52847ee42b\" (UID: \"ec8e6f24-cd09-4850-a310-1e52847ee42b\") "
Sep 16 13:01:20.922647 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:20.922536    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec8e6f24-cd09-4850-a310-1e52847ee42b-model-cache" (OuterVolumeSpecName: "model-cache") pod "ec8e6f24-cd09-4850-a310-1e52847ee42b" (UID: "ec8e6f24-cd09-4850-a310-1e52847ee42b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:01:20.922647 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:20.922585    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec8e6f24-cd09-4850-a310-1e52847ee42b-home" (OuterVolumeSpecName: "home") pod "ec8e6f24-cd09-4850-a310-1e52847ee42b" (UID: "ec8e6f24-cd09-4850-a310-1e52847ee42b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:01:20.922788 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:20.922733    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec8e6f24-cd09-4850-a310-1e52847ee42b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ec8e6f24-cd09-4850-a310-1e52847ee42b" (UID: "ec8e6f24-cd09-4850-a310-1e52847ee42b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:01:20.923022 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:20.923000    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec8e6f24-cd09-4850-a310-1e52847ee42b-model-cache\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:01:20.923022 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:20.923020    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec8e6f24-cd09-4850-a310-1e52847ee42b-home\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:01:20.923166 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:20.923049    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec8e6f24-cd09-4850-a310-1e52847ee42b-tmp-dir\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:01:20.924784 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:20.924754    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec8e6f24-cd09-4850-a310-1e52847ee42b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ec8e6f24-cd09-4850-a310-1e52847ee42b" (UID: "ec8e6f24-cd09-4850-a310-1e52847ee42b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:01:20.924881 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:20.924805    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec8e6f24-cd09-4850-a310-1e52847ee42b-dshm" (OuterVolumeSpecName: "dshm") pod "ec8e6f24-cd09-4850-a310-1e52847ee42b" (UID: "ec8e6f24-cd09-4850-a310-1e52847ee42b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:01:20.924881 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:20.924816    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec8e6f24-cd09-4850-a310-1e52847ee42b-kube-api-access-xgvhc" (OuterVolumeSpecName: "kube-api-access-xgvhc") pod "ec8e6f24-cd09-4850-a310-1e52847ee42b" (UID: "ec8e6f24-cd09-4850-a310-1e52847ee42b"). InnerVolumeSpecName "kube-api-access-xgvhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:01:21.024544 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:21.024508    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgvhc\" (UniqueName: \"kubernetes.io/projected/ec8e6f24-cd09-4850-a310-1e52847ee42b-kube-api-access-xgvhc\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:01:21.024544 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:21.024540    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec8e6f24-cd09-4850-a310-1e52847ee42b-dshm\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:01:21.024544 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:21.024550    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec8e6f24-cd09-4850-a310-1e52847ee42b-tls-certs\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:01:21.264157 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:21.264125    2577 generic.go:356] "Generic (PLEG): container finished" podID="ec8e6f24-cd09-4850-a310-1e52847ee42b" containerID="2025fcad32854ceab8f538f1cf6f6137add6be93785656ce45bb83f3a45b056f" exitCode=0
Sep 16 13:01:21.264346 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:21.264170    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-89d6bf/test-model-89d6bf-kserve-844b98bc79-xwdp2"
Sep 16 13:01:21.264346 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:21.264216    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-89d6bf/test-model-89d6bf-kserve-844b98bc79-xwdp2" event={"ID":"ec8e6f24-cd09-4850-a310-1e52847ee42b","Type":"ContainerDied","Data":"2025fcad32854ceab8f538f1cf6f6137add6be93785656ce45bb83f3a45b056f"}
Sep 16 13:01:21.264346 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:21.264270    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-89d6bf/test-model-89d6bf-kserve-844b98bc79-xwdp2" event={"ID":"ec8e6f24-cd09-4850-a310-1e52847ee42b","Type":"ContainerDied","Data":"583e167306599d65e2acf6d0af5f4e7335b811ef1394e0b956aef413e745d752"}
Sep 16 13:01:21.264346 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:21.264288    2577 scope.go:117] "RemoveContainer" containerID="2025fcad32854ceab8f538f1cf6f6137add6be93785656ce45bb83f3a45b056f"
Sep 16 13:01:21.274159 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:21.274144    2577 scope.go:117] "RemoveContainer" containerID="2025fcad32854ceab8f538f1cf6f6137add6be93785656ce45bb83f3a45b056f"
Sep 16 13:01:21.274439 ip-10-0-133-190 kubenswrapper[2577]: E0916 13:01:21.274422    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2025fcad32854ceab8f538f1cf6f6137add6be93785656ce45bb83f3a45b056f\": container with ID starting with 2025fcad32854ceab8f538f1cf6f6137add6be93785656ce45bb83f3a45b056f not found: ID does not exist" containerID="2025fcad32854ceab8f538f1cf6f6137add6be93785656ce45bb83f3a45b056f"
Sep 16 13:01:21.274500 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:21.274448    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2025fcad32854ceab8f538f1cf6f6137add6be93785656ce45bb83f3a45b056f"} err="failed to get container status \"2025fcad32854ceab8f538f1cf6f6137add6be93785656ce45bb83f3a45b056f\": rpc error: code = NotFound desc = could not find container \"2025fcad32854ceab8f538f1cf6f6137add6be93785656ce45bb83f3a45b056f\": container with ID starting with 2025fcad32854ceab8f538f1cf6f6137add6be93785656ce45bb83f3a45b056f not found: ID does not exist"
Sep 16 13:01:21.306381 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:21.306357    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-89d6bf/test-model-89d6bf-kserve-844b98bc79-xwdp2"]
Sep 16 13:01:21.311074 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:21.311053    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-89d6bf/test-model-89d6bf-kserve-844b98bc79-xwdp2"]
Sep 16 13:01:22.055690 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:22.055655    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec8e6f24-cd09-4850-a310-1e52847ee42b" path="/var/lib/kubelet/pods/ec8e6f24-cd09-4850-a310-1e52847ee42b/volumes"
Sep 16 13:01:23.216051 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:23.215996    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e5926f-7bcd585465-ms5wx"
Sep 16 13:01:23.216483 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:23.216239    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5926f-68b6695bbb-cdq2j"
Sep 16 13:01:24.764692 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:24.764657    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-89d6bf-klrdb"]
Sep 16 13:01:24.765243 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:24.765227    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec8e6f24-cd09-4850-a310-1e52847ee42b" containerName="main"
Sep 16 13:01:24.765299 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:24.765245    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec8e6f24-cd09-4850-a310-1e52847ee42b" containerName="main"
Sep 16 13:01:24.765344 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:24.765329    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec8e6f24-cd09-4850-a310-1e52847ee42b" containerName="main"
Sep 16 13:01:24.769819 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:24.769802    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-89d6bf-klrdb"
Sep 16 13:01:24.773040 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:24.773005    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-64zph\""
Sep 16 13:01:24.773487 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:24.773464    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 13:01:24.779488 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:24.779462    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-89d6bf-klrdb"]
Sep 16 13:01:24.862218 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:24.862180    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1f348b2-6d57-4e19-8ccf-dc67dda51743-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-89d6bf-klrdb\" (UID: \"b1f348b2-6d57-4e19-8ccf-dc67dda51743\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-89d6bf-klrdb"
Sep 16 13:01:24.963778 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:24.963738    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1f348b2-6d57-4e19-8ccf-dc67dda51743-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-89d6bf-klrdb\" (UID: \"b1f348b2-6d57-4e19-8ccf-dc67dda51743\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-89d6bf-klrdb"
Sep 16 13:01:24.964727 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:24.964696    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1f348b2-6d57-4e19-8ccf-dc67dda51743-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-89d6bf-klrdb\" (UID: \"b1f348b2-6d57-4e19-8ccf-dc67dda51743\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-89d6bf-klrdb"
Sep 16 13:01:25.081842 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:25.081760    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-89d6bf-klrdb"
Sep 16 13:01:25.359609 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:25.359586    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-89d6bf-klrdb"]
Sep 16 13:01:25.361283 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:01:25.361256    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1f348b2_6d57_4e19_8ccf_dc67dda51743.slice/crio-446d6c66ef32f770c0f45f1ca78c4b18e34f6736c69b51e1eea291e9edc8c6af WatchSource:0}: Error finding container 446d6c66ef32f770c0f45f1ca78c4b18e34f6736c69b51e1eea291e9edc8c6af: Status 404 returned error can't find the container with id 446d6c66ef32f770c0f45f1ca78c4b18e34f6736c69b51e1eea291e9edc8c6af
Sep 16 13:01:26.305873 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:26.305834    2577 generic.go:356] "Generic (PLEG): container finished" podID="b1f348b2-6d57-4e19-8ccf-dc67dda51743" containerID="2f5ef2541085b0c7adc73674cc5c01ac7aa601e6ff303fdd50109cc24095a608" exitCode=0
Sep 16 13:01:26.306312 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:26.305887    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-89d6bf-klrdb" event={"ID":"b1f348b2-6d57-4e19-8ccf-dc67dda51743","Type":"ContainerDied","Data":"2f5ef2541085b0c7adc73674cc5c01ac7aa601e6ff303fdd50109cc24095a608"}
Sep 16 13:01:26.306312 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:26.305941    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-89d6bf-klrdb" event={"ID":"b1f348b2-6d57-4e19-8ccf-dc67dda51743","Type":"ContainerStarted","Data":"446d6c66ef32f770c0f45f1ca78c4b18e34f6736c69b51e1eea291e9edc8c6af"}
Sep 16 13:01:27.454194 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:27.454166    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-89d6bf-klrdb"
Sep 16 13:01:27.492167 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:27.492138    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1f348b2-6d57-4e19-8ccf-dc67dda51743-maas-api-service-ca\") pod \"b1f348b2-6d57-4e19-8ccf-dc67dda51743\" (UID: \"b1f348b2-6d57-4e19-8ccf-dc67dda51743\") "
Sep 16 13:01:27.492535 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:27.492508    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1f348b2-6d57-4e19-8ccf-dc67dda51743-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b1f348b2-6d57-4e19-8ccf-dc67dda51743" (UID: "b1f348b2-6d57-4e19-8ccf-dc67dda51743"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:01:27.593855 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:27.593757    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1f348b2-6d57-4e19-8ccf-dc67dda51743-maas-api-service-ca\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:01:28.321953 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:28.321873    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-89d6bf-klrdb" event={"ID":"b1f348b2-6d57-4e19-8ccf-dc67dda51743","Type":"ContainerDied","Data":"446d6c66ef32f770c0f45f1ca78c4b18e34f6736c69b51e1eea291e9edc8c6af"}
Sep 16 13:01:28.321953 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:28.321925    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="446d6c66ef32f770c0f45f1ca78c4b18e34f6736c69b51e1eea291e9edc8c6af"
Sep 16 13:01:28.321953 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:28.321882    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-89d6bf-klrdb"
Sep 16 13:01:34.184199 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:34.184163    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 13:01:34.184636 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:34.184218    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn" podUID="05103a7a-76fc-4417-a62a-2d1439edc444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 13:01:35.844240 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:35.844203    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-89d6bf-29826060-6k7s5"]
Sep 16 13:01:35.849335 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:35.849305    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-89d6bf-29826060-6k7s5"]
Sep 16 13:01:35.889780 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:35.889749    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-89d6bf-85cfbc9694-fpzc9"]
Sep 16 13:01:35.890328 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:35.890267    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89d6bf-85cfbc9694-fpzc9" podUID="7bb299ab-9db6-44bf-820d-6711a15f9e2c" containerName="payload-pre-processing" containerID="cri-o://3921a3eca355ca8d409b8edaaaebbb3e44000c552460af988c635ef70f3f257d" gracePeriod=30
Sep 16 13:01:36.057388 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:36.057349    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66fd3e38-29c6-4824-adc1-8791f3a7c002" path="/var/lib/kubelet/pods/66fd3e38-29c6-4824-adc1-8791f3a7c002/volumes"
Sep 16 13:01:36.151219 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:36.150242    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-89d6bf-klrdb"]
Sep 16 13:01:36.153740 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:36.153474    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-89d6bf-klrdb"]
Sep 16 13:01:36.221324 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:36.221301    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89d6bf-85cfbc9694-fpzc9"
Sep 16 13:01:36.321786 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:36.321755    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bb299ab-9db6-44bf-820d-6711a15f9e2c-plugins-config-volume\") pod \"7bb299ab-9db6-44bf-820d-6711a15f9e2c\" (UID: \"7bb299ab-9db6-44bf-820d-6711a15f9e2c\") "
Sep 16 13:01:36.321957 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:36.321820    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8cx8\" (UniqueName: \"kubernetes.io/projected/7bb299ab-9db6-44bf-820d-6711a15f9e2c-kube-api-access-b8cx8\") pod \"7bb299ab-9db6-44bf-820d-6711a15f9e2c\" (UID: \"7bb299ab-9db6-44bf-820d-6711a15f9e2c\") "
Sep 16 13:01:36.322153 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:36.322133    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bb299ab-9db6-44bf-820d-6711a15f9e2c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7bb299ab-9db6-44bf-820d-6711a15f9e2c" (UID: "7bb299ab-9db6-44bf-820d-6711a15f9e2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:01:36.323994 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:36.323975    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bb299ab-9db6-44bf-820d-6711a15f9e2c-kube-api-access-b8cx8" (OuterVolumeSpecName: "kube-api-access-b8cx8") pod "7bb299ab-9db6-44bf-820d-6711a15f9e2c" (UID: "7bb299ab-9db6-44bf-820d-6711a15f9e2c"). InnerVolumeSpecName "kube-api-access-b8cx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:01:36.377687 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:36.377647    2577 generic.go:356] "Generic (PLEG): container finished" podID="7bb299ab-9db6-44bf-820d-6711a15f9e2c" containerID="3921a3eca355ca8d409b8edaaaebbb3e44000c552460af988c635ef70f3f257d" exitCode=0
Sep 16 13:01:36.377867 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:36.377703    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89d6bf-85cfbc9694-fpzc9" event={"ID":"7bb299ab-9db6-44bf-820d-6711a15f9e2c","Type":"ContainerDied","Data":"3921a3eca355ca8d409b8edaaaebbb3e44000c552460af988c635ef70f3f257d"}
Sep 16 13:01:36.377867 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:36.377725    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89d6bf-85cfbc9694-fpzc9"
Sep 16 13:01:36.377867 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:36.377758    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89d6bf-85cfbc9694-fpzc9" event={"ID":"7bb299ab-9db6-44bf-820d-6711a15f9e2c","Type":"ContainerDied","Data":"df265a7f327a87fc6dc546dcb4ceac6b3d2dc0e313e323c25633c16340889db7"}
Sep 16 13:01:36.377867 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:36.377787    2577 scope.go:117] "RemoveContainer" containerID="3921a3eca355ca8d409b8edaaaebbb3e44000c552460af988c635ef70f3f257d"
Sep 16 13:01:36.379634 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:36.379611    2577 generic.go:356] "Generic (PLEG): container finished" podID="05103a7a-76fc-4417-a62a-2d1439edc444" containerID="4597b6bb5db08ddf331310f64c8e1f1529b730ffd6466a285d45ef1b97e58640" exitCode=0
Sep 16 13:01:36.379750 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:36.379696    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn" event={"ID":"05103a7a-76fc-4417-a62a-2d1439edc444","Type":"ContainerDied","Data":"4597b6bb5db08ddf331310f64c8e1f1529b730ffd6466a285d45ef1b97e58640"}
Sep 16 13:01:36.380291 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:36.380271    2577 scope.go:117] "RemoveContainer" containerID="4597b6bb5db08ddf331310f64c8e1f1529b730ffd6466a285d45ef1b97e58640"
Sep 16 13:01:36.381612 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:36.381582    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 13:01:36.381712 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:36.381662    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 13:01:36.381712 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:36.381697    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 13:01:36.389523 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:36.389505    2577 scope.go:117] "RemoveContainer" containerID="3921a3eca355ca8d409b8edaaaebbb3e44000c552460af988c635ef70f3f257d"
Sep 16 13:01:36.389850 ip-10-0-133-190 kubenswrapper[2577]: E0916 13:01:36.389818    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3921a3eca355ca8d409b8edaaaebbb3e44000c552460af988c635ef70f3f257d\": container with ID starting with 3921a3eca355ca8d409b8edaaaebbb3e44000c552460af988c635ef70f3f257d not found: ID does not exist" containerID="3921a3eca355ca8d409b8edaaaebbb3e44000c552460af988c635ef70f3f257d"
Sep 16 13:01:36.389933 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:36.389851    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3921a3eca355ca8d409b8edaaaebbb3e44000c552460af988c635ef70f3f257d"} err="failed to get container status \"3921a3eca355ca8d409b8edaaaebbb3e44000c552460af988c635ef70f3f257d\": rpc error: code = NotFound desc = could not find container \"3921a3eca355ca8d409b8edaaaebbb3e44000c552460af988c635ef70f3f257d\": container with ID starting with 3921a3eca355ca8d409b8edaaaebbb3e44000c552460af988c635ef70f3f257d not found: ID does not exist"
Sep 16 13:01:36.423930 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:36.423874    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8cx8\" (UniqueName: \"kubernetes.io/projected/7bb299ab-9db6-44bf-820d-6711a15f9e2c-kube-api-access-b8cx8\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:01:36.423930 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:36.423906    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bb299ab-9db6-44bf-820d-6711a15f9e2c-plugins-config-volume\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:01:36.428097 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:36.427840    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-89d6bf-85cfbc9694-fpzc9"]
Sep 16 13:01:36.433401 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:36.433375    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-89d6bf-85cfbc9694-fpzc9"]
Sep 16 13:01:37.389967 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:37.389930    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn" event={"ID":"05103a7a-76fc-4417-a62a-2d1439edc444","Type":"ContainerStarted","Data":"4ad2bf2790cbf968d8fd8a69b5861c34e0666a2749c56bcebbbe98c6bb7478a2"}
Sep 16 13:01:37.600825 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:37.600796    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"
Sep 16 13:01:37.601189 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:37.601166    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Sep 16 13:01:37.601376 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:37.601346    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn" podUID="05103a7a-76fc-4417-a62a-2d1439edc444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Sep 16 13:01:38.056354 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:38.056321    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bb299ab-9db6-44bf-820d-6711a15f9e2c" path="/var/lib/kubelet/pods/7bb299ab-9db6-44bf-820d-6711a15f9e2c/volumes"
Sep 16 13:01:38.056734 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:38.056719    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1f348b2-6d57-4e19-8ccf-dc67dda51743" path="/var/lib/kubelet/pods/b1f348b2-6d57-4e19-8ccf-dc67dda51743/volumes"
Sep 16 13:01:38.601260 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:38.601225    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Sep 16 13:01:38.601922 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:38.601884    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn" podUID="05103a7a-76fc-4417-a62a-2d1439edc444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Sep 16 13:01:39.606466 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:39.606422    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Sep 16 13:01:39.606939 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:39.606491    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn" podUID="05103a7a-76fc-4417-a62a-2d1439edc444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Sep 16 13:01:40.601567 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:40.601516    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Sep 16 13:01:40.601798 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:40.601600    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn" podUID="05103a7a-76fc-4417-a62a-2d1439edc444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Sep 16 13:01:41.600910 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:41.600856    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Sep 16 13:01:41.601573 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:41.601537    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn" podUID="05103a7a-76fc-4417-a62a-2d1439edc444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Sep 16 13:01:42.600967 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:42.600929    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Sep 16 13:01:42.601578 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:42.601541    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn" podUID="05103a7a-76fc-4417-a62a-2d1439edc444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Sep 16 13:01:43.604829 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:43.604801    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"
Sep 16 13:01:43.605258 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:43.605238    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"
Sep 16 13:01:43.605847 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:43.605830    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"
Sep 16 13:01:43.756253 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:43.756223    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"]
Sep 16 13:01:43.756457 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:43.756430    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888" podUID="10ebb745-2b51-4ac4-8834-66601208abdb" containerName="istio-proxy" containerID="cri-o://44155e108f624e9db292ca763d86d9643dbb1ead640e06727dbd72aadd49522e" gracePeriod=30
Sep 16 13:01:49.030213 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.030147    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"
Sep 16 13:01:49.163567 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.163534    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10ebb745-2b51-4ac4-8834-66601208abdb-istio-podinfo\") pod \"10ebb745-2b51-4ac4-8834-66601208abdb\" (UID: \"10ebb745-2b51-4ac4-8834-66601208abdb\") "
Sep 16 13:01:49.163567 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.163570    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ebb745-2b51-4ac4-8834-66601208abdb-credential-socket\") pod \"10ebb745-2b51-4ac4-8834-66601208abdb\" (UID: \"10ebb745-2b51-4ac4-8834-66601208abdb\") "
Sep 16 13:01:49.163823 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.163605    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7jxb\" (UniqueName: \"kubernetes.io/projected/10ebb745-2b51-4ac4-8834-66601208abdb-kube-api-access-g7jxb\") pod \"10ebb745-2b51-4ac4-8834-66601208abdb\" (UID: \"10ebb745-2b51-4ac4-8834-66601208abdb\") "
Sep 16 13:01:49.163823 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.163688    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10ebb745-2b51-4ac4-8834-66601208abdb-istio-envoy\") pod \"10ebb745-2b51-4ac4-8834-66601208abdb\" (UID: \"10ebb745-2b51-4ac4-8834-66601208abdb\") "
Sep 16 13:01:49.163823 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.163730    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10ebb745-2b51-4ac4-8834-66601208abdb-istio-data\") pod \"10ebb745-2b51-4ac4-8834-66601208abdb\" (UID: \"10ebb745-2b51-4ac4-8834-66601208abdb\") "
Sep 16 13:01:49.163823 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.163772    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10ebb745-2b51-4ac4-8834-66601208abdb-istio-token\") pod \"10ebb745-2b51-4ac4-8834-66601208abdb\" (UID: \"10ebb745-2b51-4ac4-8834-66601208abdb\") "
Sep 16 13:01:49.164075 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.163848    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10ebb745-2b51-4ac4-8834-66601208abdb-workload-certs\") pod \"10ebb745-2b51-4ac4-8834-66601208abdb\" (UID: \"10ebb745-2b51-4ac4-8834-66601208abdb\") "
Sep 16 13:01:49.164075 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.163861    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10ebb745-2b51-4ac4-8834-66601208abdb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "10ebb745-2b51-4ac4-8834-66601208abdb" (UID: "10ebb745-2b51-4ac4-8834-66601208abdb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:01:49.164075 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.163896    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ebb745-2b51-4ac4-8834-66601208abdb-workload-socket\") pod \"10ebb745-2b51-4ac4-8834-66601208abdb\" (UID: \"10ebb745-2b51-4ac4-8834-66601208abdb\") "
Sep 16 13:01:49.164075 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.163925    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10ebb745-2b51-4ac4-8834-66601208abdb-istiod-ca-cert\") pod \"10ebb745-2b51-4ac4-8834-66601208abdb\" (UID: \"10ebb745-2b51-4ac4-8834-66601208abdb\") "
Sep 16 13:01:49.164295 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.164093    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10ebb745-2b51-4ac4-8834-66601208abdb-istio-data" (OuterVolumeSpecName: "istio-data") pod "10ebb745-2b51-4ac4-8834-66601208abdb" (UID: "10ebb745-2b51-4ac4-8834-66601208abdb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:01:49.164295 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.164134    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10ebb745-2b51-4ac4-8834-66601208abdb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "10ebb745-2b51-4ac4-8834-66601208abdb" (UID: "10ebb745-2b51-4ac4-8834-66601208abdb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:01:49.164517 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.164342    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10ebb745-2b51-4ac4-8834-66601208abdb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "10ebb745-2b51-4ac4-8834-66601208abdb" (UID: "10ebb745-2b51-4ac4-8834-66601208abdb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:01:49.164517 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.164398    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10ebb745-2b51-4ac4-8834-66601208abdb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "10ebb745-2b51-4ac4-8834-66601208abdb" (UID: "10ebb745-2b51-4ac4-8834-66601208abdb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:01:49.164678 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.164629    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ebb745-2b51-4ac4-8834-66601208abdb-credential-socket\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:01:49.164678 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.164654    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10ebb745-2b51-4ac4-8834-66601208abdb-istio-data\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:01:49.164678 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.164669    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10ebb745-2b51-4ac4-8834-66601208abdb-workload-certs\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:01:49.164832 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.164682    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ebb745-2b51-4ac4-8834-66601208abdb-workload-socket\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:01:49.166118 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.166087    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10ebb745-2b51-4ac4-8834-66601208abdb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "10ebb745-2b51-4ac4-8834-66601208abdb" (UID: "10ebb745-2b51-4ac4-8834-66601208abdb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 13:01:49.166239 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.166158    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10ebb745-2b51-4ac4-8834-66601208abdb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "10ebb745-2b51-4ac4-8834-66601208abdb" (UID: "10ebb745-2b51-4ac4-8834-66601208abdb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:01:49.166239 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.166200    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10ebb745-2b51-4ac4-8834-66601208abdb-istio-token" (OuterVolumeSpecName: "istio-token") pod "10ebb745-2b51-4ac4-8834-66601208abdb" (UID: "10ebb745-2b51-4ac4-8834-66601208abdb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:01:49.166320 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.166245    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10ebb745-2b51-4ac4-8834-66601208abdb-kube-api-access-g7jxb" (OuterVolumeSpecName: "kube-api-access-g7jxb") pod "10ebb745-2b51-4ac4-8834-66601208abdb" (UID: "10ebb745-2b51-4ac4-8834-66601208abdb"). InnerVolumeSpecName "kube-api-access-g7jxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:01:49.265809 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.265773    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10ebb745-2b51-4ac4-8834-66601208abdb-istio-envoy\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:01:49.265809 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.265800    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10ebb745-2b51-4ac4-8834-66601208abdb-istio-token\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:01:49.265809 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.265810    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10ebb745-2b51-4ac4-8834-66601208abdb-istiod-ca-cert\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:01:49.265809 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.265819    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10ebb745-2b51-4ac4-8834-66601208abdb-istio-podinfo\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:01:49.266201 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.265827    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7jxb\" (UniqueName: \"kubernetes.io/projected/10ebb745-2b51-4ac4-8834-66601208abdb-kube-api-access-g7jxb\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:01:49.482696 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.482524    2577 generic.go:356] "Generic (PLEG): container finished" podID="10ebb745-2b51-4ac4-8834-66601208abdb" containerID="44155e108f624e9db292ca763d86d9643dbb1ead640e06727dbd72aadd49522e" exitCode=0
Sep 16 13:01:49.482696 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.482576    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888" event={"ID":"10ebb745-2b51-4ac4-8834-66601208abdb","Type":"ContainerDied","Data":"44155e108f624e9db292ca763d86d9643dbb1ead640e06727dbd72aadd49522e"}
Sep 16 13:01:49.482696 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.482688    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888" event={"ID":"10ebb745-2b51-4ac4-8834-66601208abdb","Type":"ContainerDied","Data":"7455e21e67f53d12abcdb21f226545106333679041f3d5a5549855edc079b6ef"}
Sep 16 13:01:49.483163 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.482718    2577 scope.go:117] "RemoveContainer" containerID="44155e108f624e9db292ca763d86d9643dbb1ead640e06727dbd72aadd49522e"
Sep 16 13:01:49.483163 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.482716    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"
Sep 16 13:01:49.497406 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.497384    2577 scope.go:117] "RemoveContainer" containerID="15d28b872da26825994a5e63c24434579b6629a7aa70ab960c87994de2843d68"
Sep 16 13:01:49.507800 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.507778    2577 scope.go:117] "RemoveContainer" containerID="44155e108f624e9db292ca763d86d9643dbb1ead640e06727dbd72aadd49522e"
Sep 16 13:01:49.508160 ip-10-0-133-190 kubenswrapper[2577]: E0916 13:01:49.508141    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44155e108f624e9db292ca763d86d9643dbb1ead640e06727dbd72aadd49522e\": container with ID starting with 44155e108f624e9db292ca763d86d9643dbb1ead640e06727dbd72aadd49522e not found: ID does not exist" containerID="44155e108f624e9db292ca763d86d9643dbb1ead640e06727dbd72aadd49522e"
Sep 16 13:01:49.508242 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.508172    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44155e108f624e9db292ca763d86d9643dbb1ead640e06727dbd72aadd49522e"} err="failed to get container status \"44155e108f624e9db292ca763d86d9643dbb1ead640e06727dbd72aadd49522e\": rpc error: code = NotFound desc = could not find container \"44155e108f624e9db292ca763d86d9643dbb1ead640e06727dbd72aadd49522e\": container with ID starting with 44155e108f624e9db292ca763d86d9643dbb1ead640e06727dbd72aadd49522e not found: ID does not exist"
Sep 16 13:01:49.508242 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.508195    2577 scope.go:117] "RemoveContainer" containerID="15d28b872da26825994a5e63c24434579b6629a7aa70ab960c87994de2843d68"
Sep 16 13:01:49.508495 ip-10-0-133-190 kubenswrapper[2577]: E0916 13:01:49.508477    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15d28b872da26825994a5e63c24434579b6629a7aa70ab960c87994de2843d68\": container with ID starting with 15d28b872da26825994a5e63c24434579b6629a7aa70ab960c87994de2843d68 not found: ID does not exist" containerID="15d28b872da26825994a5e63c24434579b6629a7aa70ab960c87994de2843d68"
Sep 16 13:01:49.508539 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.508502    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15d28b872da26825994a5e63c24434579b6629a7aa70ab960c87994de2843d68"} err="failed to get container status \"15d28b872da26825994a5e63c24434579b6629a7aa70ab960c87994de2843d68\": rpc error: code = NotFound desc = could not find container \"15d28b872da26825994a5e63c24434579b6629a7aa70ab960c87994de2843d68\": container with ID starting with 15d28b872da26825994a5e63c24434579b6629a7aa70ab960c87994de2843d68 not found: ID does not exist"
Sep 16 13:01:49.513879 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.513858    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"]
Sep 16 13:01:49.519766 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.519746    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-89d6bf-openshift-default-65d86845db-mj888"]
Sep 16 13:01:49.646503 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.646472    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-524adf28-79c5d568d6-cc4ws"]
Sep 16 13:01:49.646975 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.646958    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10ebb745-2b51-4ac4-8834-66601208abdb" containerName="istio-proxy"
Sep 16 13:01:49.647084 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.646979    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="10ebb745-2b51-4ac4-8834-66601208abdb" containerName="istio-proxy"
Sep 16 13:01:49.647084 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.647000    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1f348b2-6d57-4e19-8ccf-dc67dda51743" containerName="revoke-keys"
Sep 16 13:01:49.647084 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.647008    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1f348b2-6d57-4e19-8ccf-dc67dda51743" containerName="revoke-keys"
Sep 16 13:01:49.647084 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.647052    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bb299ab-9db6-44bf-820d-6711a15f9e2c" containerName="payload-pre-processing"
Sep 16 13:01:49.647084 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.647060    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bb299ab-9db6-44bf-820d-6711a15f9e2c" containerName="payload-pre-processing"
Sep 16 13:01:49.647340 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.647167    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bb299ab-9db6-44bf-820d-6711a15f9e2c" containerName="payload-pre-processing"
Sep 16 13:01:49.647340 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.647183    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1f348b2-6d57-4e19-8ccf-dc67dda51743" containerName="revoke-keys"
Sep 16 13:01:49.647340 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.647195    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="10ebb745-2b51-4ac4-8834-66601208abdb" containerName="istio-proxy"
Sep 16 13:01:49.647340 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.647204    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="10ebb745-2b51-4ac4-8834-66601208abdb" containerName="istio-proxy"
Sep 16 13:01:49.651892 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.651871    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-524adf28-79c5d568d6-cc4ws"
Sep 16 13:01:49.655354 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.655335    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-524adf28\""
Sep 16 13:01:49.655626 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.655613    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-524adf28-dockercfg-q2f8b\""
Sep 16 13:01:49.660791 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.660771    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-524adf28-79c5d568d6-cc4ws"]
Sep 16 13:01:49.771173 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.771140    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw5xd\" (UniqueName: \"kubernetes.io/projected/cc23485f-533d-44aa-82be-d16f577d09ec-kube-api-access-bw5xd\") pod \"payload-pre-processing-e2e-mt-524adf28-79c5d568d6-cc4ws\" (UID: \"cc23485f-533d-44aa-82be-d16f577d09ec\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-524adf28-79c5d568d6-cc4ws"
Sep 16 13:01:49.771332 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.771252    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc23485f-533d-44aa-82be-d16f577d09ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-524adf28-79c5d568d6-cc4ws\" (UID: \"cc23485f-533d-44aa-82be-d16f577d09ec\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-524adf28-79c5d568d6-cc4ws"
Sep 16 13:01:49.872200 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.872168    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw5xd\" (UniqueName: \"kubernetes.io/projected/cc23485f-533d-44aa-82be-d16f577d09ec-kube-api-access-bw5xd\") pod \"payload-pre-processing-e2e-mt-524adf28-79c5d568d6-cc4ws\" (UID: \"cc23485f-533d-44aa-82be-d16f577d09ec\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-524adf28-79c5d568d6-cc4ws"
Sep 16 13:01:49.872391 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.872256    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc23485f-533d-44aa-82be-d16f577d09ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-524adf28-79c5d568d6-cc4ws\" (UID: \"cc23485f-533d-44aa-82be-d16f577d09ec\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-524adf28-79c5d568d6-cc4ws"
Sep 16 13:01:49.872972 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.872948    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc23485f-533d-44aa-82be-d16f577d09ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-524adf28-79c5d568d6-cc4ws\" (UID: \"cc23485f-533d-44aa-82be-d16f577d09ec\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-524adf28-79c5d568d6-cc4ws"
Sep 16 13:01:49.886089 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.886061    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw5xd\" (UniqueName: \"kubernetes.io/projected/cc23485f-533d-44aa-82be-d16f577d09ec-kube-api-access-bw5xd\") pod \"payload-pre-processing-e2e-mt-524adf28-79c5d568d6-cc4ws\" (UID: \"cc23485f-533d-44aa-82be-d16f577d09ec\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-524adf28-79c5d568d6-cc4ws"
Sep 16 13:01:49.963762 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:49.963729    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-524adf28-79c5d568d6-cc4ws"
Sep 16 13:01:50.056803 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:50.056768    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10ebb745-2b51-4ac4-8834-66601208abdb" path="/var/lib/kubelet/pods/10ebb745-2b51-4ac4-8834-66601208abdb/volumes"
Sep 16 13:01:50.247209 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:50.247181    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-524adf28-79c5d568d6-cc4ws"]
Sep 16 13:01:50.248238 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:01:50.248211    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc23485f_533d_44aa_82be_d16f577d09ec.slice/crio-71fb96d44d69fa849cb14ec97ac9e19e365cfbe2f43c829046c82994b0380170 WatchSource:0}: Error finding container 71fb96d44d69fa849cb14ec97ac9e19e365cfbe2f43c829046c82994b0380170: Status 404 returned error can't find the container with id 71fb96d44d69fa849cb14ec97ac9e19e365cfbe2f43c829046c82994b0380170
Sep 16 13:01:50.492834 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:50.492794    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-524adf28-79c5d568d6-cc4ws" event={"ID":"cc23485f-533d-44aa-82be-d16f577d09ec","Type":"ContainerStarted","Data":"3a8831952bf67e681d03a92ccb30f7ee56c050a8efd93c0b83b55d9643f99ffd"}
Sep 16 13:01:50.493009 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:50.492862    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-524adf28-79c5d568d6-cc4ws" event={"ID":"cc23485f-533d-44aa-82be-d16f577d09ec","Type":"ContainerStarted","Data":"71fb96d44d69fa849cb14ec97ac9e19e365cfbe2f43c829046c82994b0380170"}
Sep 16 13:01:50.493131 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:50.493077    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-524adf28-79c5d568d6-cc4ws"
Sep 16 13:01:50.515219 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:50.515170    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-524adf28-79c5d568d6-cc4ws" podStartSLOduration=1.5151548529999999 podStartE2EDuration="1.515154853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:01:50.512187848 +0000 UTC m=+1416.959420490" watchObservedRunningTime="2026-09-16 13:01:50.515154853 +0000 UTC m=+1416.962387530"
Sep 16 13:01:50.791975 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:50.791933    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1ef95c-9dd877f6f-8nkkc"]
Sep 16 13:01:50.792523 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:50.792503    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10ebb745-2b51-4ac4-8834-66601208abdb" containerName="istio-proxy"
Sep 16 13:01:50.792629 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:50.792525    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="10ebb745-2b51-4ac4-8834-66601208abdb" containerName="istio-proxy"
Sep 16 13:01:50.795938 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:50.795916    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ef95c-9dd877f6f-8nkkc"
Sep 16 13:01:50.798912 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:50.798892    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-1ef95c\""
Sep 16 13:01:50.799105 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:50.799083    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-1ef95c-dockercfg-jwk9n\""
Sep 16 13:01:50.807668 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:50.807647    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1ef95c-9dd877f6f-8nkkc"]
Sep 16 13:01:50.883600 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:50.883568    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88z2v\" (UniqueName: \"kubernetes.io/projected/0971a37b-ff6f-446c-aafb-df8e049af2db-kube-api-access-88z2v\") pod \"payload-processing-e2e-shared-a-w3-1ef95c-9dd877f6f-8nkkc\" (UID: \"0971a37b-ff6f-446c-aafb-df8e049af2db\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ef95c-9dd877f6f-8nkkc"
Sep 16 13:01:50.883761 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:50.883606    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0971a37b-ff6f-446c-aafb-df8e049af2db-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1ef95c-9dd877f6f-8nkkc\" (UID: \"0971a37b-ff6f-446c-aafb-df8e049af2db\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ef95c-9dd877f6f-8nkkc"
Sep 16 13:01:50.984636 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:50.984597    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88z2v\" (UniqueName: \"kubernetes.io/projected/0971a37b-ff6f-446c-aafb-df8e049af2db-kube-api-access-88z2v\") pod \"payload-processing-e2e-shared-a-w3-1ef95c-9dd877f6f-8nkkc\" (UID: \"0971a37b-ff6f-446c-aafb-df8e049af2db\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ef95c-9dd877f6f-8nkkc"
Sep 16 13:01:50.984636 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:50.984642    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0971a37b-ff6f-446c-aafb-df8e049af2db-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1ef95c-9dd877f6f-8nkkc\" (UID: \"0971a37b-ff6f-446c-aafb-df8e049af2db\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ef95c-9dd877f6f-8nkkc"
Sep 16 13:01:50.985420 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:50.985402    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0971a37b-ff6f-446c-aafb-df8e049af2db-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1ef95c-9dd877f6f-8nkkc\" (UID: \"0971a37b-ff6f-446c-aafb-df8e049af2db\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ef95c-9dd877f6f-8nkkc"
Sep 16 13:01:50.994232 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:50.994205    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88z2v\" (UniqueName: \"kubernetes.io/projected/0971a37b-ff6f-446c-aafb-df8e049af2db-kube-api-access-88z2v\") pod \"payload-processing-e2e-shared-a-w3-1ef95c-9dd877f6f-8nkkc\" (UID: \"0971a37b-ff6f-446c-aafb-df8e049af2db\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ef95c-9dd877f6f-8nkkc"
Sep 16 13:01:51.107703 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:51.107606    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ef95c-9dd877f6f-8nkkc"
Sep 16 13:01:51.402522 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:51.402494    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1ef95c-9dd877f6f-8nkkc"]
Sep 16 13:01:51.404429 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:01:51.404400    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0971a37b_ff6f_446c_aafb_df8e049af2db.slice/crio-605b04b4f646abd01aeb20c31616abf07e56ac4d03b9a93d26ba823bc46784f6 WatchSource:0}: Error finding container 605b04b4f646abd01aeb20c31616abf07e56ac4d03b9a93d26ba823bc46784f6: Status 404 returned error can't find the container with id 605b04b4f646abd01aeb20c31616abf07e56ac4d03b9a93d26ba823bc46784f6
Sep 16 13:01:51.505914 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:51.505881    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ef95c-9dd877f6f-8nkkc" event={"ID":"0971a37b-ff6f-446c-aafb-df8e049af2db","Type":"ContainerStarted","Data":"90969b3b92ea3983cf5b9358deedf0f7345d6a0206762764b6cff695d2726b51"}
Sep 16 13:01:51.506100 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:51.505935    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ef95c-9dd877f6f-8nkkc" event={"ID":"0971a37b-ff6f-446c-aafb-df8e049af2db","Type":"ContainerStarted","Data":"605b04b4f646abd01aeb20c31616abf07e56ac4d03b9a93d26ba823bc46784f6"}
Sep 16 13:01:51.506100 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:51.505991    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ef95c-9dd877f6f-8nkkc"
Sep 16 13:01:51.531487 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:01:51.531429    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ef95c-9dd877f6f-8nkkc" podStartSLOduration=1.5314087459999999 podStartE2EDuration="1.531408746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:01:51.529967667 +0000 UTC m=+1417.977200317" watchObservedRunningTime="2026-09-16 13:01:51.531408746 +0000 UTC m=+1417.978641390"
Sep 16 13:02:01.508349 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:01.508319    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-524adf28-79c5d568d6-cc4ws"
Sep 16 13:02:02.523129 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:02.523090    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ef95c-9dd877f6f-8nkkc"
Sep 16 13:02:07.759306 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:07.759221    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7d0256/e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l"]
Sep 16 13:02:07.763375 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:07.763351    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l"
Sep 16 13:02:07.766495 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:07.766475    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7d0256\"/\"e2e-conflict-b-32338a53-kserve-self-signed-certs\""
Sep 16 13:02:07.771457 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:07.771433    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7v58\" (UniqueName: \"kubernetes.io/projected/95d9c301-cdb6-4356-9fe1-fb91a8da0c83-kube-api-access-l7v58\") pod \"e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l\" (UID: \"95d9c301-cdb6-4356-9fe1-fb91a8da0c83\") " pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l"
Sep 16 13:02:07.771579 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:07.771499    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95d9c301-cdb6-4356-9fe1-fb91a8da0c83-home\") pod \"e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l\" (UID: \"95d9c301-cdb6-4356-9fe1-fb91a8da0c83\") " pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l"
Sep 16 13:02:07.771654 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:07.771575    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95d9c301-cdb6-4356-9fe1-fb91a8da0c83-tls-certs\") pod \"e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l\" (UID: \"95d9c301-cdb6-4356-9fe1-fb91a8da0c83\") " pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l"
Sep 16 13:02:07.771654 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:07.771647    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95d9c301-cdb6-4356-9fe1-fb91a8da0c83-model-cache\") pod \"e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l\" (UID: \"95d9c301-cdb6-4356-9fe1-fb91a8da0c83\") " pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l"
Sep 16 13:02:07.771770 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:07.771670    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95d9c301-cdb6-4356-9fe1-fb91a8da0c83-tmp-dir\") pod \"e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l\" (UID: \"95d9c301-cdb6-4356-9fe1-fb91a8da0c83\") " pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l"
Sep 16 13:02:07.771770 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:07.771696    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95d9c301-cdb6-4356-9fe1-fb91a8da0c83-dshm\") pod \"e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l\" (UID: \"95d9c301-cdb6-4356-9fe1-fb91a8da0c83\") " pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l"
Sep 16 13:02:07.774515 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:07.774489    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7d0256/e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l"]
Sep 16 13:02:07.872527 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:07.872490    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7v58\" (UniqueName: \"kubernetes.io/projected/95d9c301-cdb6-4356-9fe1-fb91a8da0c83-kube-api-access-l7v58\") pod \"e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l\" (UID: \"95d9c301-cdb6-4356-9fe1-fb91a8da0c83\") " pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l"
Sep 16 13:02:07.872672 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:07.872541    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95d9c301-cdb6-4356-9fe1-fb91a8da0c83-home\") pod \"e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l\" (UID: \"95d9c301-cdb6-4356-9fe1-fb91a8da0c83\") " pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l"
Sep 16 13:02:07.872672 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:07.872618    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95d9c301-cdb6-4356-9fe1-fb91a8da0c83-tls-certs\") pod \"e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l\" (UID: \"95d9c301-cdb6-4356-9fe1-fb91a8da0c83\") " pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l"
Sep 16 13:02:07.872787 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:07.872690    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95d9c301-cdb6-4356-9fe1-fb91a8da0c83-model-cache\") pod \"e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l\" (UID: \"95d9c301-cdb6-4356-9fe1-fb91a8da0c83\") " pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l"
Sep 16 13:02:07.872787 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:07.872713    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95d9c301-cdb6-4356-9fe1-fb91a8da0c83-tmp-dir\") pod \"e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l\" (UID: \"95d9c301-cdb6-4356-9fe1-fb91a8da0c83\") " pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l"
Sep 16 13:02:07.872787 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:07.872737    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95d9c301-cdb6-4356-9fe1-fb91a8da0c83-dshm\") pod \"e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l\" (UID: \"95d9c301-cdb6-4356-9fe1-fb91a8da0c83\") " pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l"
Sep 16 13:02:07.873086 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:07.873048    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95d9c301-cdb6-4356-9fe1-fb91a8da0c83-model-cache\") pod \"e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l\" (UID: \"95d9c301-cdb6-4356-9fe1-fb91a8da0c83\") " pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l"
Sep 16 13:02:07.873086 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:07.873016    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95d9c301-cdb6-4356-9fe1-fb91a8da0c83-home\") pod \"e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l\" (UID: \"95d9c301-cdb6-4356-9fe1-fb91a8da0c83\") " pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l"
Sep 16 13:02:07.873278 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:07.873149    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95d9c301-cdb6-4356-9fe1-fb91a8da0c83-tmp-dir\") pod \"e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l\" (UID: \"95d9c301-cdb6-4356-9fe1-fb91a8da0c83\") " pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l"
Sep 16 13:02:07.875111 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:07.875086    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95d9c301-cdb6-4356-9fe1-fb91a8da0c83-dshm\") pod \"e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l\" (UID: \"95d9c301-cdb6-4356-9fe1-fb91a8da0c83\") " pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l"
Sep 16 13:02:07.875332 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:07.875316    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95d9c301-cdb6-4356-9fe1-fb91a8da0c83-tls-certs\") pod \"e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l\" (UID: \"95d9c301-cdb6-4356-9fe1-fb91a8da0c83\") " pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l"
Sep 16 13:02:07.881990 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:07.881967    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7v58\" (UniqueName: \"kubernetes.io/projected/95d9c301-cdb6-4356-9fe1-fb91a8da0c83-kube-api-access-l7v58\") pod \"e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l\" (UID: \"95d9c301-cdb6-4356-9fe1-fb91a8da0c83\") " pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l"
Sep 16 13:02:08.073316 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:08.073226    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l"
Sep 16 13:02:08.363802 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:08.363769    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7d0256/e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l"]
Sep 16 13:02:08.365041 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:02:08.364995    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95d9c301_cdb6_4356_9fe1_fb91a8da0c83.slice/crio-2efcdd3766414b7fd62324c5f469a7a80194d87455a7f4e01f6509baad1d7704 WatchSource:0}: Error finding container 2efcdd3766414b7fd62324c5f469a7a80194d87455a7f4e01f6509baad1d7704: Status 404 returned error can't find the container with id 2efcdd3766414b7fd62324c5f469a7a80194d87455a7f4e01f6509baad1d7704
Sep 16 13:02:08.622318 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:08.622228    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l" event={"ID":"95d9c301-cdb6-4356-9fe1-fb91a8da0c83","Type":"ContainerStarted","Data":"d3de0af8cd65275251fb9bf30528fc8e52ab8bd0bf7d1c4769ee3b383d949c83"}
Sep 16 13:02:08.622318 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:08.622277    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l" event={"ID":"95d9c301-cdb6-4356-9fe1-fb91a8da0c83","Type":"ContainerStarted","Data":"2efcdd3766414b7fd62324c5f469a7a80194d87455a7f4e01f6509baad1d7704"}
Sep 16 13:02:08.649639 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:08.649593    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l" podStartSLOduration=1.649578668 podStartE2EDuration="1.649578668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:02:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:02:08.648112843 +0000 UTC m=+1435.095345485" watchObservedRunningTime="2026-09-16 13:02:08.649578668 +0000 UTC m=+1435.096811309"
Sep 16 13:02:14.731538 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:14.731506    2577 scope.go:117] "RemoveContainer" containerID="1a66bb79e44e4a03ffaffdd1bb6463b7f5e83f72663c7138be70f2226eb1622f"
Sep 16 13:02:18.074154 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:18.074118    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l"
Sep 16 13:02:18.074522 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:18.074187    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l"
Sep 16 13:02:18.086526 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:18.086504    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l"
Sep 16 13:02:18.699136 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:18.699110    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l"
Sep 16 13:02:23.231705 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:23.231651    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e344f9-85bcc9d6c-p4xzc"]
Sep 16 13:02:23.238957 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:23.238894    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e344f9-85bcc9d6c-p4xzc"
Sep 16 13:02:23.242310 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:23.242274    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-e344f9\""
Sep 16 13:02:23.242493 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:23.242459    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-e344f9-dockercfg-mdbpm\""
Sep 16 13:02:23.249287 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:23.249262    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e344f9-85bcc9d6c-p4xzc"]
Sep 16 13:02:23.350643 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:23.350610    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkhv8\" (UniqueName: \"kubernetes.io/projected/8258857c-15b7-443d-91e2-97e0ed2eb413-kube-api-access-tkhv8\") pod \"payload-processing-e2e-shared-b-w3-e344f9-85bcc9d6c-p4xzc\" (UID: \"8258857c-15b7-443d-91e2-97e0ed2eb413\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e344f9-85bcc9d6c-p4xzc"
Sep 16 13:02:23.350822 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:23.350685    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8258857c-15b7-443d-91e2-97e0ed2eb413-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-e344f9-85bcc9d6c-p4xzc\" (UID: \"8258857c-15b7-443d-91e2-97e0ed2eb413\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e344f9-85bcc9d6c-p4xzc"
Sep 16 13:02:23.451612 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:23.451582    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkhv8\" (UniqueName: \"kubernetes.io/projected/8258857c-15b7-443d-91e2-97e0ed2eb413-kube-api-access-tkhv8\") pod \"payload-processing-e2e-shared-b-w3-e344f9-85bcc9d6c-p4xzc\" (UID: \"8258857c-15b7-443d-91e2-97e0ed2eb413\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e344f9-85bcc9d6c-p4xzc"
Sep 16 13:02:23.451612 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:23.451626    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8258857c-15b7-443d-91e2-97e0ed2eb413-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-e344f9-85bcc9d6c-p4xzc\" (UID: \"8258857c-15b7-443d-91e2-97e0ed2eb413\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e344f9-85bcc9d6c-p4xzc"
Sep 16 13:02:23.452385 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:23.452352    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8258857c-15b7-443d-91e2-97e0ed2eb413-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-e344f9-85bcc9d6c-p4xzc\" (UID: \"8258857c-15b7-443d-91e2-97e0ed2eb413\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e344f9-85bcc9d6c-p4xzc"
Sep 16 13:02:23.462623 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:23.462593    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkhv8\" (UniqueName: \"kubernetes.io/projected/8258857c-15b7-443d-91e2-97e0ed2eb413-kube-api-access-tkhv8\") pod \"payload-processing-e2e-shared-b-w3-e344f9-85bcc9d6c-p4xzc\" (UID: \"8258857c-15b7-443d-91e2-97e0ed2eb413\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e344f9-85bcc9d6c-p4xzc"
Sep 16 13:02:23.555558 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:23.555522    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e344f9-85bcc9d6c-p4xzc"
Sep 16 13:02:23.847853 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:23.847827    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e344f9-85bcc9d6c-p4xzc"]
Sep 16 13:02:23.850340 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:02:23.850312    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8258857c_15b7_443d_91e2_97e0ed2eb413.slice/crio-096ac588861cb7c3f3eda02de2ec0a7b816649c570aa5830876b9bdbc1a10ee7 WatchSource:0}: Error finding container 096ac588861cb7c3f3eda02de2ec0a7b816649c570aa5830876b9bdbc1a10ee7: Status 404 returned error can't find the container with id 096ac588861cb7c3f3eda02de2ec0a7b816649c570aa5830876b9bdbc1a10ee7
Sep 16 13:02:24.724816 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:24.724760    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e344f9-85bcc9d6c-p4xzc" event={"ID":"8258857c-15b7-443d-91e2-97e0ed2eb413","Type":"ContainerStarted","Data":"32ec150b106c8675f49b2a35943967678d1d3cd0afbb6d56c394626ed69e29af"}
Sep 16 13:02:24.725291 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:24.724918    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e344f9-85bcc9d6c-p4xzc"
Sep 16 13:02:24.725291 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:24.724936    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e344f9-85bcc9d6c-p4xzc" event={"ID":"8258857c-15b7-443d-91e2-97e0ed2eb413","Type":"ContainerStarted","Data":"096ac588861cb7c3f3eda02de2ec0a7b816649c570aa5830876b9bdbc1a10ee7"}
Sep 16 13:02:24.745920 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:24.745872    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e344f9-85bcc9d6c-p4xzc" podStartSLOduration=1.7458587730000001 podStartE2EDuration="1.745858773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:02:24.744331155 +0000 UTC m=+1451.191563798" watchObservedRunningTime="2026-09-16 13:02:24.745858773 +0000 UTC m=+1451.193091418"
Sep 16 13:02:35.734203 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:35.734173    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e344f9-85bcc9d6c-p4xzc"
Sep 16 13:02:46.843288 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:46.843256    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-490797/distinct-2-w0-490797-kserve-75844fdfc8-2g87x"]
Sep 16 13:02:46.846935 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:46.846919    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-490797/distinct-2-w0-490797-kserve-75844fdfc8-2g87x"
Sep 16 13:02:46.850500 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:46.850478    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-490797\"/\"distinct-2-w0-490797-kserve-self-signed-certs\""
Sep 16 13:02:46.857669 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:46.857647    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-490797/distinct-2-w0-490797-kserve-75844fdfc8-2g87x"]
Sep 16 13:02:46.901476 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:46.901441    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34141f56-6c73-4ac9-adc6-0da953fca281-dshm\") pod \"distinct-2-w0-490797-kserve-75844fdfc8-2g87x\" (UID: \"34141f56-6c73-4ac9-adc6-0da953fca281\") " pod="e2e-models-e2e-worker-w0-490797/distinct-2-w0-490797-kserve-75844fdfc8-2g87x"
Sep 16 13:02:46.901476 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:46.901475    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wdf6\" (UniqueName: \"kubernetes.io/projected/34141f56-6c73-4ac9-adc6-0da953fca281-kube-api-access-2wdf6\") pod \"distinct-2-w0-490797-kserve-75844fdfc8-2g87x\" (UID: \"34141f56-6c73-4ac9-adc6-0da953fca281\") " pod="e2e-models-e2e-worker-w0-490797/distinct-2-w0-490797-kserve-75844fdfc8-2g87x"
Sep 16 13:02:46.901672 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:46.901553    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34141f56-6c73-4ac9-adc6-0da953fca281-tls-certs\") pod \"distinct-2-w0-490797-kserve-75844fdfc8-2g87x\" (UID: \"34141f56-6c73-4ac9-adc6-0da953fca281\") " pod="e2e-models-e2e-worker-w0-490797/distinct-2-w0-490797-kserve-75844fdfc8-2g87x"
Sep 16 13:02:46.901672 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:46.901608    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34141f56-6c73-4ac9-adc6-0da953fca281-tmp-dir\") pod \"distinct-2-w0-490797-kserve-75844fdfc8-2g87x\" (UID: \"34141f56-6c73-4ac9-adc6-0da953fca281\") " pod="e2e-models-e2e-worker-w0-490797/distinct-2-w0-490797-kserve-75844fdfc8-2g87x"
Sep 16 13:02:46.901672 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:46.901644    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34141f56-6c73-4ac9-adc6-0da953fca281-model-cache\") pod \"distinct-2-w0-490797-kserve-75844fdfc8-2g87x\" (UID: \"34141f56-6c73-4ac9-adc6-0da953fca281\") " pod="e2e-models-e2e-worker-w0-490797/distinct-2-w0-490797-kserve-75844fdfc8-2g87x"
Sep 16 13:02:46.901770 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:46.901706    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34141f56-6c73-4ac9-adc6-0da953fca281-home\") pod \"distinct-2-w0-490797-kserve-75844fdfc8-2g87x\" (UID: \"34141f56-6c73-4ac9-adc6-0da953fca281\") " pod="e2e-models-e2e-worker-w0-490797/distinct-2-w0-490797-kserve-75844fdfc8-2g87x"
Sep 16 13:02:47.002948 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:47.002915    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34141f56-6c73-4ac9-adc6-0da953fca281-home\") pod \"distinct-2-w0-490797-kserve-75844fdfc8-2g87x\" (UID: \"34141f56-6c73-4ac9-adc6-0da953fca281\") " pod="e2e-models-e2e-worker-w0-490797/distinct-2-w0-490797-kserve-75844fdfc8-2g87x"
Sep 16 13:02:47.003159 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:47.002959    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34141f56-6c73-4ac9-adc6-0da953fca281-dshm\") pod \"distinct-2-w0-490797-kserve-75844fdfc8-2g87x\" (UID: \"34141f56-6c73-4ac9-adc6-0da953fca281\") " pod="e2e-models-e2e-worker-w0-490797/distinct-2-w0-490797-kserve-75844fdfc8-2g87x"
Sep 16 13:02:47.003159 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:47.002977    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wdf6\" (UniqueName: \"kubernetes.io/projected/34141f56-6c73-4ac9-adc6-0da953fca281-kube-api-access-2wdf6\") pod \"distinct-2-w0-490797-kserve-75844fdfc8-2g87x\" (UID: \"34141f56-6c73-4ac9-adc6-0da953fca281\") " pod="e2e-models-e2e-worker-w0-490797/distinct-2-w0-490797-kserve-75844fdfc8-2g87x"
Sep 16 13:02:47.003159 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:47.003120    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34141f56-6c73-4ac9-adc6-0da953fca281-tls-certs\") pod \"distinct-2-w0-490797-kserve-75844fdfc8-2g87x\" (UID: \"34141f56-6c73-4ac9-adc6-0da953fca281\") " pod="e2e-models-e2e-worker-w0-490797/distinct-2-w0-490797-kserve-75844fdfc8-2g87x"
Sep 16 13:02:47.003326 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:47.003168    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34141f56-6c73-4ac9-adc6-0da953fca281-tmp-dir\") pod \"distinct-2-w0-490797-kserve-75844fdfc8-2g87x\" (UID: \"34141f56-6c73-4ac9-adc6-0da953fca281\") " pod="e2e-models-e2e-worker-w0-490797/distinct-2-w0-490797-kserve-75844fdfc8-2g87x"
Sep 16 13:02:47.003326 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:47.003215    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34141f56-6c73-4ac9-adc6-0da953fca281-model-cache\") pod \"distinct-2-w0-490797-kserve-75844fdfc8-2g87x\" (UID: \"34141f56-6c73-4ac9-adc6-0da953fca281\") " pod="e2e-models-e2e-worker-w0-490797/distinct-2-w0-490797-kserve-75844fdfc8-2g87x"
Sep 16 13:02:47.003434 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:47.003357    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34141f56-6c73-4ac9-adc6-0da953fca281-home\") pod \"distinct-2-w0-490797-kserve-75844fdfc8-2g87x\" (UID: \"34141f56-6c73-4ac9-adc6-0da953fca281\") " pod="e2e-models-e2e-worker-w0-490797/distinct-2-w0-490797-kserve-75844fdfc8-2g87x"
Sep 16 13:02:47.003563 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:47.003539    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34141f56-6c73-4ac9-adc6-0da953fca281-tmp-dir\") pod \"distinct-2-w0-490797-kserve-75844fdfc8-2g87x\" (UID: \"34141f56-6c73-4ac9-adc6-0da953fca281\") " pod="e2e-models-e2e-worker-w0-490797/distinct-2-w0-490797-kserve-75844fdfc8-2g87x"
Sep 16 13:02:47.003620 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:47.003576    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34141f56-6c73-4ac9-adc6-0da953fca281-model-cache\") pod \"distinct-2-w0-490797-kserve-75844fdfc8-2g87x\" (UID: \"34141f56-6c73-4ac9-adc6-0da953fca281\") " pod="e2e-models-e2e-worker-w0-490797/distinct-2-w0-490797-kserve-75844fdfc8-2g87x"
Sep 16 13:02:47.005455 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:47.005433    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34141f56-6c73-4ac9-adc6-0da953fca281-dshm\") pod \"distinct-2-w0-490797-kserve-75844fdfc8-2g87x\" (UID: \"34141f56-6c73-4ac9-adc6-0da953fca281\") " pod="e2e-models-e2e-worker-w0-490797/distinct-2-w0-490797-kserve-75844fdfc8-2g87x"
Sep 16 13:02:47.005714 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:47.005694    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34141f56-6c73-4ac9-adc6-0da953fca281-tls-certs\") pod \"distinct-2-w0-490797-kserve-75844fdfc8-2g87x\" (UID: \"34141f56-6c73-4ac9-adc6-0da953fca281\") " pod="e2e-models-e2e-worker-w0-490797/distinct-2-w0-490797-kserve-75844fdfc8-2g87x"
Sep 16 13:02:47.016456 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:47.016420    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wdf6\" (UniqueName: \"kubernetes.io/projected/34141f56-6c73-4ac9-adc6-0da953fca281-kube-api-access-2wdf6\") pod \"distinct-2-w0-490797-kserve-75844fdfc8-2g87x\" (UID: \"34141f56-6c73-4ac9-adc6-0da953fca281\") " pod="e2e-models-e2e-worker-w0-490797/distinct-2-w0-490797-kserve-75844fdfc8-2g87x"
Sep 16 13:02:47.157138 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:47.157057    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-490797/distinct-2-w0-490797-kserve-75844fdfc8-2g87x"
Sep 16 13:02:47.455306 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:47.455281    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-490797/distinct-2-w0-490797-kserve-75844fdfc8-2g87x"]
Sep 16 13:02:47.457130 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:02:47.457104    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34141f56_6c73_4ac9_adc6_0da953fca281.slice/crio-2625ab95f7b505ac24a5251729c03ac51e07787aa8762fb0f09e2f63c61e028b WatchSource:0}: Error finding container 2625ab95f7b505ac24a5251729c03ac51e07787aa8762fb0f09e2f63c61e028b: Status 404 returned error can't find the container with id 2625ab95f7b505ac24a5251729c03ac51e07787aa8762fb0f09e2f63c61e028b
Sep 16 13:02:47.876978 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:47.876945    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-490797/distinct-2-w0-490797-kserve-75844fdfc8-2g87x" event={"ID":"34141f56-6c73-4ac9-adc6-0da953fca281","Type":"ContainerStarted","Data":"edf14b9ac7d93e4d913807775e1321764f82317862f8a8ba8d6730a31752a602"}
Sep 16 13:02:47.877421 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:47.876986    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-490797/distinct-2-w0-490797-kserve-75844fdfc8-2g87x" event={"ID":"34141f56-6c73-4ac9-adc6-0da953fca281","Type":"ContainerStarted","Data":"2625ab95f7b505ac24a5251729c03ac51e07787aa8762fb0f09e2f63c61e028b"}
Sep 16 13:02:47.915488 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:47.915441    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-490797/distinct-2-w0-490797-kserve-75844fdfc8-2g87x" podStartSLOduration=1.915424631 podStartE2EDuration="1.915424631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:02:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:02:47.914008537 +0000 UTC m=+1474.361241180" watchObservedRunningTime="2026-09-16 13:02:47.915424631 +0000 UTC m=+1474.362657269"
Sep 16 13:02:57.158222 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:57.158175    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-490797/distinct-2-w0-490797-kserve-75844fdfc8-2g87x"
Sep 16 13:02:57.158599 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:57.158239    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-490797/distinct-2-w0-490797-kserve-75844fdfc8-2g87x"
Sep 16 13:02:57.170850 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:57.170824    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-490797/distinct-2-w0-490797-kserve-75844fdfc8-2g87x"
Sep 16 13:02:57.970106 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:02:57.970078    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-490797/distinct-2-w0-490797-kserve-75844fdfc8-2g87x"
Sep 16 13:03:01.754843 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:01.754811    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"]
Sep 16 13:03:01.755315 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:01.755073    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv" podUID="898bb80d-221e-419f-a1c7-d9a28ced4e99" containerName="istio-proxy" containerID="cri-o://b3a4cf81d668ae35fd2669f3a0cdddfe5751fe14bf633229c656b3ce9b58ec10" gracePeriod=30
Sep 16 13:03:03.044393 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:03.044353    2577 patch_prober.go:28] interesting pod/e2e-noten-b159da-openshift-default-788696c96-2mmrv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 13:03:03.044776 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:03.044427    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv" podUID="898bb80d-221e-419f-a1c7-d9a28ced4e99" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:03:07.014606 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:07.014580    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"
Sep 16 13:03:07.023995 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:07.023926    2577 generic.go:356] "Generic (PLEG): container finished" podID="898bb80d-221e-419f-a1c7-d9a28ced4e99" containerID="b3a4cf81d668ae35fd2669f3a0cdddfe5751fe14bf633229c656b3ce9b58ec10" exitCode=0
Sep 16 13:03:07.023995 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:07.023973    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"
Sep 16 13:03:07.024211 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:07.024020    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv" event={"ID":"898bb80d-221e-419f-a1c7-d9a28ced4e99","Type":"ContainerDied","Data":"b3a4cf81d668ae35fd2669f3a0cdddfe5751fe14bf633229c656b3ce9b58ec10"}
Sep 16 13:03:07.024211 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:07.024145    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv" event={"ID":"898bb80d-221e-419f-a1c7-d9a28ced4e99","Type":"ContainerDied","Data":"0256468f0cd585edebd5a1a666d35c3b5328c0e0a0adccb3959384d9811c4fd5"}
Sep 16 13:03:07.024211 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:07.024177    2577 scope.go:117] "RemoveContainer" containerID="b3a4cf81d668ae35fd2669f3a0cdddfe5751fe14bf633229c656b3ce9b58ec10"
Sep 16 13:03:07.033798 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:07.033781    2577 scope.go:117] "RemoveContainer" containerID="b3a4cf81d668ae35fd2669f3a0cdddfe5751fe14bf633229c656b3ce9b58ec10"
Sep 16 13:03:07.034094 ip-10-0-133-190 kubenswrapper[2577]: E0916 13:03:07.034068    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3a4cf81d668ae35fd2669f3a0cdddfe5751fe14bf633229c656b3ce9b58ec10\": container with ID starting with b3a4cf81d668ae35fd2669f3a0cdddfe5751fe14bf633229c656b3ce9b58ec10 not found: ID does not exist" containerID="b3a4cf81d668ae35fd2669f3a0cdddfe5751fe14bf633229c656b3ce9b58ec10"
Sep 16 13:03:07.034196 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:07.034101    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3a4cf81d668ae35fd2669f3a0cdddfe5751fe14bf633229c656b3ce9b58ec10"} err="failed to get container status \"b3a4cf81d668ae35fd2669f3a0cdddfe5751fe14bf633229c656b3ce9b58ec10\": rpc error: code = NotFound desc = could not find container \"b3a4cf81d668ae35fd2669f3a0cdddfe5751fe14bf633229c656b3ce9b58ec10\": container with ID starting with b3a4cf81d668ae35fd2669f3a0cdddfe5751fe14bf633229c656b3ce9b58ec10 not found: ID does not exist"
Sep 16 13:03:07.112419 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:07.112395    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/898bb80d-221e-419f-a1c7-d9a28ced4e99-istio-envoy\") pod \"898bb80d-221e-419f-a1c7-d9a28ced4e99\" (UID: \"898bb80d-221e-419f-a1c7-d9a28ced4e99\") "
Sep 16 13:03:07.112616 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:07.112440    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/898bb80d-221e-419f-a1c7-d9a28ced4e99-workload-certs\") pod \"898bb80d-221e-419f-a1c7-d9a28ced4e99\" (UID: \"898bb80d-221e-419f-a1c7-d9a28ced4e99\") "
Sep 16 13:03:07.112616 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:07.112471    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/898bb80d-221e-419f-a1c7-d9a28ced4e99-istio-token\") pod \"898bb80d-221e-419f-a1c7-d9a28ced4e99\" (UID: \"898bb80d-221e-419f-a1c7-d9a28ced4e99\") "
Sep 16 13:03:07.112616 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:07.112507    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/898bb80d-221e-419f-a1c7-d9a28ced4e99-credential-socket\") pod \"898bb80d-221e-419f-a1c7-d9a28ced4e99\" (UID: \"898bb80d-221e-419f-a1c7-d9a28ced4e99\") "
Sep 16 13:03:07.112616 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:07.112526    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/898bb80d-221e-419f-a1c7-d9a28ced4e99-istio-podinfo\") pod \"898bb80d-221e-419f-a1c7-d9a28ced4e99\" (UID: \"898bb80d-221e-419f-a1c7-d9a28ced4e99\") "
Sep 16 13:03:07.112616 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:07.112548    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/898bb80d-221e-419f-a1c7-d9a28ced4e99-istiod-ca-cert\") pod \"898bb80d-221e-419f-a1c7-d9a28ced4e99\" (UID: \"898bb80d-221e-419f-a1c7-d9a28ced4e99\") "
Sep 16 13:03:07.112616 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:07.112610    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9xjf\" (UniqueName: \"kubernetes.io/projected/898bb80d-221e-419f-a1c7-d9a28ced4e99-kube-api-access-w9xjf\") pod \"898bb80d-221e-419f-a1c7-d9a28ced4e99\" (UID: \"898bb80d-221e-419f-a1c7-d9a28ced4e99\") "
Sep 16 13:03:07.113669 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:07.112650    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/898bb80d-221e-419f-a1c7-d9a28ced4e99-workload-socket\") pod \"898bb80d-221e-419f-a1c7-d9a28ced4e99\" (UID: \"898bb80d-221e-419f-a1c7-d9a28ced4e99\") "
Sep 16 13:03:07.113669 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:07.112785    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/898bb80d-221e-419f-a1c7-d9a28ced4e99-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "898bb80d-221e-419f-a1c7-d9a28ced4e99" (UID: "898bb80d-221e-419f-a1c7-d9a28ced4e99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:03:07.113669 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:07.113169    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/898bb80d-221e-419f-a1c7-d9a28ced4e99-istio-data\") pod \"898bb80d-221e-419f-a1c7-d9a28ced4e99\" (UID: \"898bb80d-221e-419f-a1c7-d9a28ced4e99\") "
Sep 16 13:03:07.113669 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:07.113166    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/898bb80d-221e-419f-a1c7-d9a28ced4e99-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "898bb80d-221e-419f-a1c7-d9a28ced4e99" (UID: "898bb80d-221e-419f-a1c7-d9a28ced4e99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:03:07.113669 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:07.113370    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/898bb80d-221e-419f-a1c7-d9a28ced4e99-istio-data" (OuterVolumeSpecName: "istio-data") pod "898bb80d-221e-419f-a1c7-d9a28ced4e99" (UID: "898bb80d-221e-419f-a1c7-d9a28ced4e99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:03:07.113669 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:07.113265    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/898bb80d-221e-419f-a1c7-d9a28ced4e99-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "898bb80d-221e-419f-a1c7-d9a28ced4e99" (UID: "898bb80d-221e-419f-a1c7-d9a28ced4e99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:03:07.113977 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:07.113829    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/898bb80d-221e-419f-a1c7-d9a28ced4e99-workload-socket\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:03:07.113977 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:07.113852    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/898bb80d-221e-419f-a1c7-d9a28ced4e99-istio-data\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:03:07.113977 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:07.113867    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/898bb80d-221e-419f-a1c7-d9a28ced4e99-workload-certs\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:03:07.113977 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:07.113880    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/898bb80d-221e-419f-a1c7-d9a28ced4e99-istiod-ca-cert\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:03:07.113977 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:07.113949    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/898bb80d-221e-419f-a1c7-d9a28ced4e99-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "898bb80d-221e-419f-a1c7-d9a28ced4e99" (UID: "898bb80d-221e-419f-a1c7-d9a28ced4e99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:03:07.115198 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:07.115157    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/898bb80d-221e-419f-a1c7-d9a28ced4e99-kube-api-access-w9xjf" (OuterVolumeSpecName: "kube-api-access-w9xjf") pod "898bb80d-221e-419f-a1c7-d9a28ced4e99" (UID: "898bb80d-221e-419f-a1c7-d9a28ced4e99"). InnerVolumeSpecName "kube-api-access-w9xjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:03:07.115297 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:07.115205    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/898bb80d-221e-419f-a1c7-d9a28ced4e99-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "898bb80d-221e-419f-a1c7-d9a28ced4e99" (UID: "898bb80d-221e-419f-a1c7-d9a28ced4e99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:03:07.115680 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:07.115654    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/898bb80d-221e-419f-a1c7-d9a28ced4e99-istio-token" (OuterVolumeSpecName: "istio-token") pod "898bb80d-221e-419f-a1c7-d9a28ced4e99" (UID: "898bb80d-221e-419f-a1c7-d9a28ced4e99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:03:07.115848 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:07.115827    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/898bb80d-221e-419f-a1c7-d9a28ced4e99-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "898bb80d-221e-419f-a1c7-d9a28ced4e99" (UID: "898bb80d-221e-419f-a1c7-d9a28ced4e99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 13:03:07.215502 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:07.215469    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/898bb80d-221e-419f-a1c7-d9a28ced4e99-istio-token\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:03:07.215502 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:07.215501    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/898bb80d-221e-419f-a1c7-d9a28ced4e99-credential-socket\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:03:07.215502 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:07.215512    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/898bb80d-221e-419f-a1c7-d9a28ced4e99-istio-podinfo\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:03:07.215735 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:07.215520    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9xjf\" (UniqueName: \"kubernetes.io/projected/898bb80d-221e-419f-a1c7-d9a28ced4e99-kube-api-access-w9xjf\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:03:07.215735 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:07.215529    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/898bb80d-221e-419f-a1c7-d9a28ced4e99-istio-envoy\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:03:07.350568 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:07.350533    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"]
Sep 16 13:03:07.357092 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:07.357066    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-b159da-openshift-default-788696c96-2mmrv"]
Sep 16 13:03:08.055199 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:08.055172    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="898bb80d-221e-419f-a1c7-d9a28ced4e99" path="/var/lib/kubelet/pods/898bb80d-221e-419f-a1c7-d9a28ced4e99/volumes"
Sep 16 13:03:14.148854 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:14.148823    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vbt8d_0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174/console-operator/1.log"
Sep 16 13:03:14.154453 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:14.154428    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vbt8d_0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174/console-operator/1.log"
Sep 16 13:03:14.770888 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:14.770857    2577 scope.go:117] "RemoveContainer" containerID="4557f9176cb29640260a579f1448633d35227028bc72b389174147c2dbf26af4"
Sep 16 13:03:43.357461 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:43.357383    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-1ef95c/auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft"]
Sep 16 13:03:43.357924 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:43.357907    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="898bb80d-221e-419f-a1c7-d9a28ced4e99" containerName="istio-proxy"
Sep 16 13:03:43.357969 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:43.357926    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="898bb80d-221e-419f-a1c7-d9a28ced4e99" containerName="istio-proxy"
Sep 16 13:03:43.358075 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:43.358063    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="898bb80d-221e-419f-a1c7-d9a28ced4e99" containerName="istio-proxy"
Sep 16 13:03:43.362589 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:43.362574    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1ef95c/auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft"
Sep 16 13:03:43.366068 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:43.366009    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1ef95c\"/\"default-dockercfg-58kpl\""
Sep 16 13:03:43.367313 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:43.367292    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1ef95c\"/\"openshift-service-ca.crt\""
Sep 16 13:03:43.367441 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:43.367294    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1ef95c\"/\"kube-root-ca.crt\""
Sep 16 13:03:43.367441 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:43.367291    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1ef95c\"/\"auth-test-model-1ef95c-kserve-self-signed-certs\""
Sep 16 13:03:43.373465 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:43.373446    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1ef95c/auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft"]
Sep 16 13:03:43.483262 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:43.483232    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jppj4\" (UniqueName: \"kubernetes.io/projected/6ddfa698-b491-497a-be0c-602cb4c1f8ed-kube-api-access-jppj4\") pod \"auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft\" (UID: \"6ddfa698-b491-497a-be0c-602cb4c1f8ed\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft"
Sep 16 13:03:43.483262 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:43.483272    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ddfa698-b491-497a-be0c-602cb4c1f8ed-home\") pod \"auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft\" (UID: \"6ddfa698-b491-497a-be0c-602cb4c1f8ed\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft"
Sep 16 13:03:43.483520 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:43.483299    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ddfa698-b491-497a-be0c-602cb4c1f8ed-dshm\") pod \"auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft\" (UID: \"6ddfa698-b491-497a-be0c-602cb4c1f8ed\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft"
Sep 16 13:03:43.483520 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:43.483416    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ddfa698-b491-497a-be0c-602cb4c1f8ed-model-cache\") pod \"auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft\" (UID: \"6ddfa698-b491-497a-be0c-602cb4c1f8ed\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft"
Sep 16 13:03:43.483520 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:43.483462    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ddfa698-b491-497a-be0c-602cb4c1f8ed-tmp-dir\") pod \"auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft\" (UID: \"6ddfa698-b491-497a-be0c-602cb4c1f8ed\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft"
Sep 16 13:03:43.483647 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:43.483539    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ddfa698-b491-497a-be0c-602cb4c1f8ed-tls-certs\") pod \"auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft\" (UID: \"6ddfa698-b491-497a-be0c-602cb4c1f8ed\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft"
Sep 16 13:03:43.584230 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:43.584199    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ddfa698-b491-497a-be0c-602cb4c1f8ed-model-cache\") pod \"auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft\" (UID: \"6ddfa698-b491-497a-be0c-602cb4c1f8ed\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft"
Sep 16 13:03:43.584379 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:43.584236    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ddfa698-b491-497a-be0c-602cb4c1f8ed-tmp-dir\") pod \"auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft\" (UID: \"6ddfa698-b491-497a-be0c-602cb4c1f8ed\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft"
Sep 16 13:03:43.584379 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:43.584267    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ddfa698-b491-497a-be0c-602cb4c1f8ed-tls-certs\") pod \"auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft\" (UID: \"6ddfa698-b491-497a-be0c-602cb4c1f8ed\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft"
Sep 16 13:03:43.584379 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:43.584336    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jppj4\" (UniqueName: \"kubernetes.io/projected/6ddfa698-b491-497a-be0c-602cb4c1f8ed-kube-api-access-jppj4\") pod \"auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft\" (UID: \"6ddfa698-b491-497a-be0c-602cb4c1f8ed\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft"
Sep 16 13:03:43.584379 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:43.584363    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ddfa698-b491-497a-be0c-602cb4c1f8ed-home\") pod \"auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft\" (UID: \"6ddfa698-b491-497a-be0c-602cb4c1f8ed\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft"
Sep 16 13:03:43.584573 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:43.584400    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ddfa698-b491-497a-be0c-602cb4c1f8ed-dshm\") pod \"auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft\" (UID: \"6ddfa698-b491-497a-be0c-602cb4c1f8ed\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft"
Sep 16 13:03:43.584642 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:43.584617    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ddfa698-b491-497a-be0c-602cb4c1f8ed-model-cache\") pod \"auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft\" (UID: \"6ddfa698-b491-497a-be0c-602cb4c1f8ed\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft"
Sep 16 13:03:43.584705 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:43.584637    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ddfa698-b491-497a-be0c-602cb4c1f8ed-tmp-dir\") pod \"auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft\" (UID: \"6ddfa698-b491-497a-be0c-602cb4c1f8ed\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft"
Sep 16 13:03:43.584788 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:43.584766    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ddfa698-b491-497a-be0c-602cb4c1f8ed-home\") pod \"auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft\" (UID: \"6ddfa698-b491-497a-be0c-602cb4c1f8ed\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft"
Sep 16 13:03:43.586730 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:43.586700    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ddfa698-b491-497a-be0c-602cb4c1f8ed-dshm\") pod \"auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft\" (UID: \"6ddfa698-b491-497a-be0c-602cb4c1f8ed\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft"
Sep 16 13:03:43.587005 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:43.586984    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ddfa698-b491-497a-be0c-602cb4c1f8ed-tls-certs\") pod \"auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft\" (UID: \"6ddfa698-b491-497a-be0c-602cb4c1f8ed\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft"
Sep 16 13:03:43.598818 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:43.598790    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jppj4\" (UniqueName: \"kubernetes.io/projected/6ddfa698-b491-497a-be0c-602cb4c1f8ed-kube-api-access-jppj4\") pod \"auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft\" (UID: \"6ddfa698-b491-497a-be0c-602cb4c1f8ed\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft"
Sep 16 13:03:43.673579 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:43.673484    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1ef95c/auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft"
Sep 16 13:03:43.957279 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:43.957251    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1ef95c/auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft"]
Sep 16 13:03:43.959329 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:03:43.959301    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ddfa698_b491_497a_be0c_602cb4c1f8ed.slice/crio-f6c102ea1258ef73f68498ee0271d2310698c56c9b1a543e8501155bfaeda7de WatchSource:0}: Error finding container f6c102ea1258ef73f68498ee0271d2310698c56c9b1a543e8501155bfaeda7de: Status 404 returned error can't find the container with id f6c102ea1258ef73f68498ee0271d2310698c56c9b1a543e8501155bfaeda7de
Sep 16 13:03:44.265597 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:44.265557    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1ef95c/auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft" event={"ID":"6ddfa698-b491-497a-be0c-602cb4c1f8ed","Type":"ContainerStarted","Data":"798fd6dd202f00e32f384a02fc6d9b852d70e07888ad52ed27de00c4c34e5b31"}
Sep 16 13:03:44.265771 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:44.265608    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1ef95c/auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft" event={"ID":"6ddfa698-b491-497a-be0c-602cb4c1f8ed","Type":"ContainerStarted","Data":"f6c102ea1258ef73f68498ee0271d2310698c56c9b1a543e8501155bfaeda7de"}
Sep 16 13:03:44.288398 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:44.288341    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-1ef95c/auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft" podStartSLOduration=1.288320696 podStartE2EDuration="1.288320696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:03:44.285392468 +0000 UTC m=+1530.732625120" watchObservedRunningTime="2026-09-16 13:03:44.288320696 +0000 UTC m=+1530.735553340"
Sep 16 13:03:53.673893 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:53.673852    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-1ef95c/auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft"
Sep 16 13:03:53.674318 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:53.673912    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-1ef95c/auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft"
Sep 16 13:03:53.686779 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:53.686755    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-1ef95c/auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft"
Sep 16 13:03:54.339444 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:03:54.339410    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-1ef95c/auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft"
Sep 16 13:04:12.550242 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:12.550205    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-490797-lcgjb"]
Sep 16 13:04:12.554319 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:12.554302    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-490797-lcgjb"
Sep 16 13:04:12.557471 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:12.557452    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-64zph\""
Sep 16 13:04:12.558458 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:12.558443    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 13:04:12.572445 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:12.572420    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-490797-lcgjb"]
Sep 16 13:04:12.682175 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:12.682140    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9741c7d0-a898-4b78-a0a6-e32288c5a78b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-490797-lcgjb\" (UID: \"9741c7d0-a898-4b78-a0a6-e32288c5a78b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-490797-lcgjb"
Sep 16 13:04:12.783568 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:12.783534    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9741c7d0-a898-4b78-a0a6-e32288c5a78b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-490797-lcgjb\" (UID: \"9741c7d0-a898-4b78-a0a6-e32288c5a78b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-490797-lcgjb"
Sep 16 13:04:12.784199 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:12.784173    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9741c7d0-a898-4b78-a0a6-e32288c5a78b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-490797-lcgjb\" (UID: \"9741c7d0-a898-4b78-a0a6-e32288c5a78b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-490797-lcgjb"
Sep 16 13:04:12.865478 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:12.865399    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-490797-lcgjb"
Sep 16 13:04:13.141392 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:13.141365    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-490797-lcgjb"]
Sep 16 13:04:13.144213 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:04:13.144181    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9741c7d0_a898_4b78_a0a6_e32288c5a78b.slice/crio-c9ed09885668c80d85fb13cbef2acdf5a94faaf15a5b90215cec9817c3543aa4 WatchSource:0}: Error finding container c9ed09885668c80d85fb13cbef2acdf5a94faaf15a5b90215cec9817c3543aa4: Status 404 returned error can't find the container with id c9ed09885668c80d85fb13cbef2acdf5a94faaf15a5b90215cec9817c3543aa4
Sep 16 13:04:13.451331 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:13.451292    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-490797-lcgjb" event={"ID":"9741c7d0-a898-4b78-a0a6-e32288c5a78b","Type":"ContainerStarted","Data":"ef1e64ac850f62711beea6b04a8450f4580e19e9e3e04900e842cf40a62e1cd6"}
Sep 16 13:04:13.451481 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:13.451357    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-490797-lcgjb" event={"ID":"9741c7d0-a898-4b78-a0a6-e32288c5a78b","Type":"ContainerStarted","Data":"c9ed09885668c80d85fb13cbef2acdf5a94faaf15a5b90215cec9817c3543aa4"}
Sep 16 13:04:13.474948 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:13.474894    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-490797-lcgjb" podStartSLOduration=1.474877441 podStartE2EDuration="1.474877441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:04:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:04:13.472517495 +0000 UTC m=+1559.919750138" watchObservedRunningTime="2026-09-16 13:04:13.474877441 +0000 UTC m=+1559.922110084"
Sep 16 13:04:14.459297 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:14.459255    2577 generic.go:356] "Generic (PLEG): container finished" podID="9741c7d0-a898-4b78-a0a6-e32288c5a78b" containerID="ef1e64ac850f62711beea6b04a8450f4580e19e9e3e04900e842cf40a62e1cd6" exitCode=0
Sep 16 13:04:14.459785 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:14.459307    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-490797-lcgjb" event={"ID":"9741c7d0-a898-4b78-a0a6-e32288c5a78b","Type":"ContainerDied","Data":"ef1e64ac850f62711beea6b04a8450f4580e19e9e3e04900e842cf40a62e1cd6"}
Sep 16 13:04:14.801097 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:14.801066    2577 scope.go:117] "RemoveContainer" containerID="bd8a8f121404aa1e858a2447f7dc0604f9d209fe0ba51ae7255234a6751af61a"
Sep 16 13:04:14.811056 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:14.811008    2577 scope.go:117] "RemoveContainer" containerID="fb2cb5a28df9d9099392e5f855128f1c166941bb9e3d56911a0f3ab361bffd06"
Sep 16 13:04:15.561231 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:15.561198    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7d0256/e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l"]
Sep 16 13:04:15.612779 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:15.612756    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-490797-lcgjb"
Sep 16 13:04:15.713932 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:15.713904    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9741c7d0-a898-4b78-a0a6-e32288c5a78b-maas-api-service-ca\") pod \"9741c7d0-a898-4b78-a0a6-e32288c5a78b\" (UID: \"9741c7d0-a898-4b78-a0a6-e32288c5a78b\") "
Sep 16 13:04:15.714218 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:15.714197    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9741c7d0-a898-4b78-a0a6-e32288c5a78b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9741c7d0-a898-4b78-a0a6-e32288c5a78b" (UID: "9741c7d0-a898-4b78-a0a6-e32288c5a78b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:04:15.815589 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:15.815557    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9741c7d0-a898-4b78-a0a6-e32288c5a78b-maas-api-service-ca\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:04:16.484867 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:16.484827    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-490797-lcgjb" event={"ID":"9741c7d0-a898-4b78-a0a6-e32288c5a78b","Type":"ContainerDied","Data":"c9ed09885668c80d85fb13cbef2acdf5a94faaf15a5b90215cec9817c3543aa4"}
Sep 16 13:04:16.485091 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:16.484919    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9ed09885668c80d85fb13cbef2acdf5a94faaf15a5b90215cec9817c3543aa4"
Sep 16 13:04:16.485091 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:16.484843    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-490797-lcgjb"
Sep 16 13:04:17.365442 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:17.365410    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-490797/distinct-2-w0-490797-kserve-75844fdfc8-2g87x"]
Sep 16 13:04:17.396023 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:17.395989    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-490797/simulator-w0-490797-kserve-7fb7c79cf5-w9scf"]
Sep 16 13:04:28.424607 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:28.424573    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-490797-7c9447cf5d-gb8wg"]
Sep 16 13:04:28.424998 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:28.424817    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-490797-7c9447cf5d-gb8wg" podUID="bb83e365-cc67-44bc-9a58-f1eb36ff6232" containerName="payload-pre-processing" containerID="cri-o://4f1f15a6d57177724c50e840ca3c8791c652b9a3e8746bc53c3b880b14961d60" gracePeriod=30
Sep 16 13:04:28.517727 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:28.517690    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-490797-7c9447cf5d-gb8wg" podUID="bb83e365-cc67-44bc-9a58-f1eb36ff6232" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.49:9004: connect: connection refused"
Sep 16 13:04:28.562274 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:28.562241    2577 generic.go:356] "Generic (PLEG): container finished" podID="bb83e365-cc67-44bc-9a58-f1eb36ff6232" containerID="4f1f15a6d57177724c50e840ca3c8791c652b9a3e8746bc53c3b880b14961d60" exitCode=0
Sep 16 13:04:28.562439 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:28.562294    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-490797-7c9447cf5d-gb8wg" event={"ID":"bb83e365-cc67-44bc-9a58-f1eb36ff6232","Type":"ContainerDied","Data":"4f1f15a6d57177724c50e840ca3c8791c652b9a3e8746bc53c3b880b14961d60"}
Sep 16 13:04:28.659884 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:28.659846    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-490797-lcgjb"]
Sep 16 13:04:28.663445 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:28.663419    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-490797-lcgjb"]
Sep 16 13:04:28.729597 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:28.729571    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-490797-7c9447cf5d-gb8wg"
Sep 16 13:04:28.770703 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:28.770670    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbzr6\" (UniqueName: \"kubernetes.io/projected/bb83e365-cc67-44bc-9a58-f1eb36ff6232-kube-api-access-rbzr6\") pod \"bb83e365-cc67-44bc-9a58-f1eb36ff6232\" (UID: \"bb83e365-cc67-44bc-9a58-f1eb36ff6232\") "
Sep 16 13:04:28.770906 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:28.770807    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb83e365-cc67-44bc-9a58-f1eb36ff6232-plugins-config-volume\") pod \"bb83e365-cc67-44bc-9a58-f1eb36ff6232\" (UID: \"bb83e365-cc67-44bc-9a58-f1eb36ff6232\") "
Sep 16 13:04:28.771345 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:28.771321    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb83e365-cc67-44bc-9a58-f1eb36ff6232-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bb83e365-cc67-44bc-9a58-f1eb36ff6232" (UID: "bb83e365-cc67-44bc-9a58-f1eb36ff6232"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:04:28.773289 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:28.773258    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb83e365-cc67-44bc-9a58-f1eb36ff6232-kube-api-access-rbzr6" (OuterVolumeSpecName: "kube-api-access-rbzr6") pod "bb83e365-cc67-44bc-9a58-f1eb36ff6232" (UID: "bb83e365-cc67-44bc-9a58-f1eb36ff6232"). InnerVolumeSpecName "kube-api-access-rbzr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:04:28.872867 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:28.872836    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb83e365-cc67-44bc-9a58-f1eb36ff6232-plugins-config-volume\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:04:28.872867 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:28.872861    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbzr6\" (UniqueName: \"kubernetes.io/projected/bb83e365-cc67-44bc-9a58-f1eb36ff6232-kube-api-access-rbzr6\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:04:29.569581 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:29.569537    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-490797-7c9447cf5d-gb8wg" event={"ID":"bb83e365-cc67-44bc-9a58-f1eb36ff6232","Type":"ContainerDied","Data":"f3bc9bc81062b10709f19e2a269e6a0a505b5a8099887f92e028b7cd2ac238ac"}
Sep 16 13:04:29.569581 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:29.569573    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-490797-7c9447cf5d-gb8wg"
Sep 16 13:04:29.570190 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:29.569617    2577 scope.go:117] "RemoveContainer" containerID="4f1f15a6d57177724c50e840ca3c8791c652b9a3e8746bc53c3b880b14961d60"
Sep 16 13:04:29.593725 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:29.593697    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-490797-7c9447cf5d-gb8wg"]
Sep 16 13:04:29.598592 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:29.598570    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-490797-7c9447cf5d-gb8wg"]
Sep 16 13:04:30.056154 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:30.056115    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9741c7d0-a898-4b78-a0a6-e32288c5a78b" path="/var/lib/kubelet/pods/9741c7d0-a898-4b78-a0a6-e32288c5a78b/volumes"
Sep 16 13:04:30.056617 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:30.056595    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb83e365-cc67-44bc-9a58-f1eb36ff6232" path="/var/lib/kubelet/pods/bb83e365-cc67-44bc-9a58-f1eb36ff6232/volumes"
Sep 16 13:04:30.584441 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:30.584385    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l" podUID="95d9c301-cdb6-4356-9fe1-fb91a8da0c83" containerName="main" containerID="cri-o://d3de0af8cd65275251fb9bf30528fc8e52ab8bd0bf7d1c4769ee3b383d949c83" gracePeriod=45
Sep 16 13:04:30.845080 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:30.845005    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l"
Sep 16 13:04:30.891976 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:30.891943    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95d9c301-cdb6-4356-9fe1-fb91a8da0c83-model-cache\") pod \"95d9c301-cdb6-4356-9fe1-fb91a8da0c83\" (UID: \"95d9c301-cdb6-4356-9fe1-fb91a8da0c83\") "
Sep 16 13:04:30.892198 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:30.892070    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95d9c301-cdb6-4356-9fe1-fb91a8da0c83-tmp-dir\") pod \"95d9c301-cdb6-4356-9fe1-fb91a8da0c83\" (UID: \"95d9c301-cdb6-4356-9fe1-fb91a8da0c83\") "
Sep 16 13:04:30.892198 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:30.892108    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95d9c301-cdb6-4356-9fe1-fb91a8da0c83-tls-certs\") pod \"95d9c301-cdb6-4356-9fe1-fb91a8da0c83\" (UID: \"95d9c301-cdb6-4356-9fe1-fb91a8da0c83\") "
Sep 16 13:04:30.892198 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:30.892144    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7v58\" (UniqueName: \"kubernetes.io/projected/95d9c301-cdb6-4356-9fe1-fb91a8da0c83-kube-api-access-l7v58\") pod \"95d9c301-cdb6-4356-9fe1-fb91a8da0c83\" (UID: \"95d9c301-cdb6-4356-9fe1-fb91a8da0c83\") "
Sep 16 13:04:30.892361 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:30.892227    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95d9c301-cdb6-4356-9fe1-fb91a8da0c83-dshm\") pod \"95d9c301-cdb6-4356-9fe1-fb91a8da0c83\" (UID: \"95d9c301-cdb6-4356-9fe1-fb91a8da0c83\") "
Sep 16 13:04:30.892361 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:30.892302    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95d9c301-cdb6-4356-9fe1-fb91a8da0c83-home\") pod \"95d9c301-cdb6-4356-9fe1-fb91a8da0c83\" (UID: \"95d9c301-cdb6-4356-9fe1-fb91a8da0c83\") "
Sep 16 13:04:30.893063 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:30.892547    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95d9c301-cdb6-4356-9fe1-fb91a8da0c83-model-cache" (OuterVolumeSpecName: "model-cache") pod "95d9c301-cdb6-4356-9fe1-fb91a8da0c83" (UID: "95d9c301-cdb6-4356-9fe1-fb91a8da0c83"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:04:30.893063 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:30.892728    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95d9c301-cdb6-4356-9fe1-fb91a8da0c83-model-cache\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:04:30.893063 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:30.892845    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95d9c301-cdb6-4356-9fe1-fb91a8da0c83-home" (OuterVolumeSpecName: "home") pod "95d9c301-cdb6-4356-9fe1-fb91a8da0c83" (UID: "95d9c301-cdb6-4356-9fe1-fb91a8da0c83"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:04:30.893063 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:30.892991    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95d9c301-cdb6-4356-9fe1-fb91a8da0c83-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "95d9c301-cdb6-4356-9fe1-fb91a8da0c83" (UID: "95d9c301-cdb6-4356-9fe1-fb91a8da0c83"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:04:30.895442 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:30.895410    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95d9c301-cdb6-4356-9fe1-fb91a8da0c83-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "95d9c301-cdb6-4356-9fe1-fb91a8da0c83" (UID: "95d9c301-cdb6-4356-9fe1-fb91a8da0c83"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:04:30.895635 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:30.895611    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95d9c301-cdb6-4356-9fe1-fb91a8da0c83-dshm" (OuterVolumeSpecName: "dshm") pod "95d9c301-cdb6-4356-9fe1-fb91a8da0c83" (UID: "95d9c301-cdb6-4356-9fe1-fb91a8da0c83"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:04:30.895730 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:30.895644    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95d9c301-cdb6-4356-9fe1-fb91a8da0c83-kube-api-access-l7v58" (OuterVolumeSpecName: "kube-api-access-l7v58") pod "95d9c301-cdb6-4356-9fe1-fb91a8da0c83" (UID: "95d9c301-cdb6-4356-9fe1-fb91a8da0c83"). InnerVolumeSpecName "kube-api-access-l7v58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:04:30.993733 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:30.993697    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95d9c301-cdb6-4356-9fe1-fb91a8da0c83-tmp-dir\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:04:30.993733 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:30.993724    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95d9c301-cdb6-4356-9fe1-fb91a8da0c83-tls-certs\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:04:30.993733 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:30.993733    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7v58\" (UniqueName: \"kubernetes.io/projected/95d9c301-cdb6-4356-9fe1-fb91a8da0c83-kube-api-access-l7v58\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:04:30.993733 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:30.993742    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95d9c301-cdb6-4356-9fe1-fb91a8da0c83-dshm\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:04:30.993998 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:30.993751    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95d9c301-cdb6-4356-9fe1-fb91a8da0c83-home\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:04:31.584933 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:31.584899    2577 generic.go:356] "Generic (PLEG): container finished" podID="95d9c301-cdb6-4356-9fe1-fb91a8da0c83" containerID="d3de0af8cd65275251fb9bf30528fc8e52ab8bd0bf7d1c4769ee3b383d949c83" exitCode=0
Sep 16 13:04:31.585376 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:31.584941    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l"
Sep 16 13:04:31.585376 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:31.584994    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l" event={"ID":"95d9c301-cdb6-4356-9fe1-fb91a8da0c83","Type":"ContainerDied","Data":"d3de0af8cd65275251fb9bf30528fc8e52ab8bd0bf7d1c4769ee3b383d949c83"}
Sep 16 13:04:31.585376 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:31.585059    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l" event={"ID":"95d9c301-cdb6-4356-9fe1-fb91a8da0c83","Type":"ContainerDied","Data":"2efcdd3766414b7fd62324c5f469a7a80194d87455a7f4e01f6509baad1d7704"}
Sep 16 13:04:31.585376 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:31.585085    2577 scope.go:117] "RemoveContainer" containerID="d3de0af8cd65275251fb9bf30528fc8e52ab8bd0bf7d1c4769ee3b383d949c83"
Sep 16 13:04:31.598139 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:31.598118    2577 scope.go:117] "RemoveContainer" containerID="d3de0af8cd65275251fb9bf30528fc8e52ab8bd0bf7d1c4769ee3b383d949c83"
Sep 16 13:04:31.598452 ip-10-0-133-190 kubenswrapper[2577]: E0916 13:04:31.598434    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3de0af8cd65275251fb9bf30528fc8e52ab8bd0bf7d1c4769ee3b383d949c83\": container with ID starting with d3de0af8cd65275251fb9bf30528fc8e52ab8bd0bf7d1c4769ee3b383d949c83 not found: ID does not exist" containerID="d3de0af8cd65275251fb9bf30528fc8e52ab8bd0bf7d1c4769ee3b383d949c83"
Sep 16 13:04:31.598554 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:31.598462    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3de0af8cd65275251fb9bf30528fc8e52ab8bd0bf7d1c4769ee3b383d949c83"} err="failed to get container status \"d3de0af8cd65275251fb9bf30528fc8e52ab8bd0bf7d1c4769ee3b383d949c83\": rpc error: code = NotFound desc = could not find container \"d3de0af8cd65275251fb9bf30528fc8e52ab8bd0bf7d1c4769ee3b383d949c83\": container with ID starting with d3de0af8cd65275251fb9bf30528fc8e52ab8bd0bf7d1c4769ee3b383d949c83 not found: ID does not exist"
Sep 16 13:04:31.611760 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:31.611736    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7d0256/e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l"]
Sep 16 13:04:31.617689 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:31.617669    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7d0256/e2e-conflict-b-32338a53-kserve-77bbb64467-q4j7l"]
Sep 16 13:04:32.056416 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.056386    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95d9c301-cdb6-4356-9fe1-fb91a8da0c83" path="/var/lib/kubelet/pods/95d9c301-cdb6-4356-9fe1-fb91a8da0c83/volumes"
Sep 16 13:04:32.386039 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.385937    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-490797/distinct-2-w0-490797-kserve-75844fdfc8-2g87x" podUID="34141f56-6c73-4ac9-adc6-0da953fca281" containerName="main" containerID="cri-o://edf14b9ac7d93e4d913807775e1321764f82317862f8a8ba8d6730a31752a602" gracePeriod=45
Sep 16 13:04:32.415803 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.415773    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-490797/simulator-w0-490797-kserve-7fb7c79cf5-w9scf" podUID="9742c8d6-9162-460b-8426-ecbde47a077a" containerName="main" containerID="cri-o://d1b95dfed751cf34f4aaf52368d8a24fab172cd4c48f2870f45ddbdf2ed40008" gracePeriod=45
Sep 16 13:04:32.600517 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.600488    2577 generic.go:356] "Generic (PLEG): container finished" podID="9742c8d6-9162-460b-8426-ecbde47a077a" containerID="d1b95dfed751cf34f4aaf52368d8a24fab172cd4c48f2870f45ddbdf2ed40008" exitCode=0
Sep 16 13:04:32.600950 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.600551    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-490797/simulator-w0-490797-kserve-7fb7c79cf5-w9scf" event={"ID":"9742c8d6-9162-460b-8426-ecbde47a077a","Type":"ContainerDied","Data":"d1b95dfed751cf34f4aaf52368d8a24fab172cd4c48f2870f45ddbdf2ed40008"}
Sep 16 13:04:32.604005 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.603973    2577 generic.go:356] "Generic (PLEG): container finished" podID="34141f56-6c73-4ac9-adc6-0da953fca281" containerID="edf14b9ac7d93e4d913807775e1321764f82317862f8a8ba8d6730a31752a602" exitCode=0
Sep 16 13:04:32.604152 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.604057    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-490797/distinct-2-w0-490797-kserve-75844fdfc8-2g87x" event={"ID":"34141f56-6c73-4ac9-adc6-0da953fca281","Type":"ContainerDied","Data":"edf14b9ac7d93e4d913807775e1321764f82317862f8a8ba8d6730a31752a602"}
Sep 16 13:04:32.720160 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.720084    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-490797/distinct-2-w0-490797-kserve-75844fdfc8-2g87x"
Sep 16 13:04:32.723571 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.723547    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-490797/simulator-w0-490797-kserve-7fb7c79cf5-w9scf"
Sep 16 13:04:32.813104 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.813071    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9742c8d6-9162-460b-8426-ecbde47a077a-home\") pod \"9742c8d6-9162-460b-8426-ecbde47a077a\" (UID: \"9742c8d6-9162-460b-8426-ecbde47a077a\") "
Sep 16 13:04:32.813294 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.813116    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wdf6\" (UniqueName: \"kubernetes.io/projected/34141f56-6c73-4ac9-adc6-0da953fca281-kube-api-access-2wdf6\") pod \"34141f56-6c73-4ac9-adc6-0da953fca281\" (UID: \"34141f56-6c73-4ac9-adc6-0da953fca281\") "
Sep 16 13:04:32.813294 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.813144    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34141f56-6c73-4ac9-adc6-0da953fca281-tmp-dir\") pod \"34141f56-6c73-4ac9-adc6-0da953fca281\" (UID: \"34141f56-6c73-4ac9-adc6-0da953fca281\") "
Sep 16 13:04:32.813491 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.813363    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9742c8d6-9162-460b-8426-ecbde47a077a-home" (OuterVolumeSpecName: "home") pod "9742c8d6-9162-460b-8426-ecbde47a077a" (UID: "9742c8d6-9162-460b-8426-ecbde47a077a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:04:32.813491 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.813479    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34141f56-6c73-4ac9-adc6-0da953fca281-tls-certs\") pod \"34141f56-6c73-4ac9-adc6-0da953fca281\" (UID: \"34141f56-6c73-4ac9-adc6-0da953fca281\") "
Sep 16 13:04:32.813632 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.813383    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34141f56-6c73-4ac9-adc6-0da953fca281-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "34141f56-6c73-4ac9-adc6-0da953fca281" (UID: "34141f56-6c73-4ac9-adc6-0da953fca281"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:04:32.813632 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.813511    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9742c8d6-9162-460b-8426-ecbde47a077a-model-cache\") pod \"9742c8d6-9162-460b-8426-ecbde47a077a\" (UID: \"9742c8d6-9162-460b-8426-ecbde47a077a\") "
Sep 16 13:04:32.813632 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.813537    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8vw7\" (UniqueName: \"kubernetes.io/projected/9742c8d6-9162-460b-8426-ecbde47a077a-kube-api-access-b8vw7\") pod \"9742c8d6-9162-460b-8426-ecbde47a077a\" (UID: \"9742c8d6-9162-460b-8426-ecbde47a077a\") "
Sep 16 13:04:32.814104 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.814075    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9742c8d6-9162-460b-8426-ecbde47a077a-model-cache" (OuterVolumeSpecName: "model-cache") pod "9742c8d6-9162-460b-8426-ecbde47a077a" (UID: "9742c8d6-9162-460b-8426-ecbde47a077a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:04:32.814210 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.814133    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9742c8d6-9162-460b-8426-ecbde47a077a-tmp-dir\") pod \"9742c8d6-9162-460b-8426-ecbde47a077a\" (UID: \"9742c8d6-9162-460b-8426-ecbde47a077a\") "
Sep 16 13:04:32.814210 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.814178    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9742c8d6-9162-460b-8426-ecbde47a077a-tls-certs\") pod \"9742c8d6-9162-460b-8426-ecbde47a077a\" (UID: \"9742c8d6-9162-460b-8426-ecbde47a077a\") "
Sep 16 13:04:32.814316 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.814242    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34141f56-6c73-4ac9-adc6-0da953fca281-dshm\") pod \"34141f56-6c73-4ac9-adc6-0da953fca281\" (UID: \"34141f56-6c73-4ac9-adc6-0da953fca281\") "
Sep 16 13:04:32.814316 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.814267    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34141f56-6c73-4ac9-adc6-0da953fca281-home\") pod \"34141f56-6c73-4ac9-adc6-0da953fca281\" (UID: \"34141f56-6c73-4ac9-adc6-0da953fca281\") "
Sep 16 13:04:32.814316 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.814298    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9742c8d6-9162-460b-8426-ecbde47a077a-dshm\") pod \"9742c8d6-9162-460b-8426-ecbde47a077a\" (UID: \"9742c8d6-9162-460b-8426-ecbde47a077a\") "
Sep 16 13:04:32.814462 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.814349    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34141f56-6c73-4ac9-adc6-0da953fca281-model-cache\") pod \"34141f56-6c73-4ac9-adc6-0da953fca281\" (UID: \"34141f56-6c73-4ac9-adc6-0da953fca281\") "
Sep 16 13:04:32.814462 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.814374    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9742c8d6-9162-460b-8426-ecbde47a077a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9742c8d6-9162-460b-8426-ecbde47a077a" (UID: "9742c8d6-9162-460b-8426-ecbde47a077a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:04:32.814855 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.814830    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34141f56-6c73-4ac9-adc6-0da953fca281-home" (OuterVolumeSpecName: "home") pod "34141f56-6c73-4ac9-adc6-0da953fca281" (UID: "34141f56-6c73-4ac9-adc6-0da953fca281"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:04:32.817719 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.814889    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9742c8d6-9162-460b-8426-ecbde47a077a-home\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:04:32.817719 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.816609    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34141f56-6c73-4ac9-adc6-0da953fca281-tmp-dir\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:04:32.817719 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.816630    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9742c8d6-9162-460b-8426-ecbde47a077a-model-cache\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:04:32.817719 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.816656    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9742c8d6-9162-460b-8426-ecbde47a077a-tmp-dir\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:04:32.817719 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.816868    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34141f56-6c73-4ac9-adc6-0da953fca281-model-cache" (OuterVolumeSpecName: "model-cache") pod "34141f56-6c73-4ac9-adc6-0da953fca281" (UID: "34141f56-6c73-4ac9-adc6-0da953fca281"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:04:32.827488 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.818092    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34141f56-6c73-4ac9-adc6-0da953fca281-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "34141f56-6c73-4ac9-adc6-0da953fca281" (UID: "34141f56-6c73-4ac9-adc6-0da953fca281"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:04:32.827488 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.818128    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34141f56-6c73-4ac9-adc6-0da953fca281-kube-api-access-2wdf6" (OuterVolumeSpecName: "kube-api-access-2wdf6") pod "34141f56-6c73-4ac9-adc6-0da953fca281" (UID: "34141f56-6c73-4ac9-adc6-0da953fca281"). InnerVolumeSpecName "kube-api-access-2wdf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:04:32.827488 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.819392    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34141f56-6c73-4ac9-adc6-0da953fca281-dshm" (OuterVolumeSpecName: "dshm") pod "34141f56-6c73-4ac9-adc6-0da953fca281" (UID: "34141f56-6c73-4ac9-adc6-0da953fca281"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:04:32.832188 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.830389    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9742c8d6-9162-460b-8426-ecbde47a077a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9742c8d6-9162-460b-8426-ecbde47a077a" (UID: "9742c8d6-9162-460b-8426-ecbde47a077a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:04:32.832714 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.832688    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9742c8d6-9162-460b-8426-ecbde47a077a-dshm" (OuterVolumeSpecName: "dshm") pod "9742c8d6-9162-460b-8426-ecbde47a077a" (UID: "9742c8d6-9162-460b-8426-ecbde47a077a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:04:32.833293 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.833269    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9742c8d6-9162-460b-8426-ecbde47a077a-kube-api-access-b8vw7" (OuterVolumeSpecName: "kube-api-access-b8vw7") pod "9742c8d6-9162-460b-8426-ecbde47a077a" (UID: "9742c8d6-9162-460b-8426-ecbde47a077a"). InnerVolumeSpecName "kube-api-access-b8vw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:04:32.917606 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.917570    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34141f56-6c73-4ac9-adc6-0da953fca281-tls-certs\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:04:32.917606 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.917601    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8vw7\" (UniqueName: \"kubernetes.io/projected/9742c8d6-9162-460b-8426-ecbde47a077a-kube-api-access-b8vw7\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:04:32.917606 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.917611    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9742c8d6-9162-460b-8426-ecbde47a077a-tls-certs\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:04:32.917846 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.917619    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34141f56-6c73-4ac9-adc6-0da953fca281-dshm\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:04:32.917846 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.917626    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34141f56-6c73-4ac9-adc6-0da953fca281-home\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:04:32.917846 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.917633    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9742c8d6-9162-460b-8426-ecbde47a077a-dshm\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:04:32.917846 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.917641    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34141f56-6c73-4ac9-adc6-0da953fca281-model-cache\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:04:32.917846 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:32.917649    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wdf6\" (UniqueName: \"kubernetes.io/projected/34141f56-6c73-4ac9-adc6-0da953fca281-kube-api-access-2wdf6\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:04:33.612662 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:33.612633    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-490797/simulator-w0-490797-kserve-7fb7c79cf5-w9scf"
Sep 16 13:04:33.612662 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:33.612632    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-490797/simulator-w0-490797-kserve-7fb7c79cf5-w9scf" event={"ID":"9742c8d6-9162-460b-8426-ecbde47a077a","Type":"ContainerDied","Data":"95d0a7709404c53ca21b511a73aa8568c7aa9880403fdebd1db742ab2fccab82"}
Sep 16 13:04:33.613185 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:33.612712    2577 scope.go:117] "RemoveContainer" containerID="d1b95dfed751cf34f4aaf52368d8a24fab172cd4c48f2870f45ddbdf2ed40008"
Sep 16 13:04:33.614324 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:33.614236    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-490797/distinct-2-w0-490797-kserve-75844fdfc8-2g87x"
Sep 16 13:04:33.614428 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:33.614306    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-490797/distinct-2-w0-490797-kserve-75844fdfc8-2g87x" event={"ID":"34141f56-6c73-4ac9-adc6-0da953fca281","Type":"ContainerDied","Data":"2625ab95f7b505ac24a5251729c03ac51e07787aa8762fb0f09e2f63c61e028b"}
Sep 16 13:04:33.624223 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:33.624158    2577 scope.go:117] "RemoveContainer" containerID="edf14b9ac7d93e4d913807775e1321764f82317862f8a8ba8d6730a31752a602"
Sep 16 13:04:33.643081 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:33.643058    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-490797/simulator-w0-490797-kserve-7fb7c79cf5-w9scf"]
Sep 16 13:04:33.647374 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:33.647354    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-490797/simulator-w0-490797-kserve-7fb7c79cf5-w9scf"]
Sep 16 13:04:33.659828 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:33.659776    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-490797/distinct-2-w0-490797-kserve-75844fdfc8-2g87x"]
Sep 16 13:04:33.664457 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:33.664437    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-490797/distinct-2-w0-490797-kserve-75844fdfc8-2g87x"]
Sep 16 13:04:34.057750 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:34.057719    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34141f56-6c73-4ac9-adc6-0da953fca281" path="/var/lib/kubelet/pods/34141f56-6c73-4ac9-adc6-0da953fca281/volumes"
Sep 16 13:04:34.058142 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:34.058128    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9742c8d6-9162-460b-8426-ecbde47a077a" path="/var/lib/kubelet/pods/9742c8d6-9162-460b-8426-ecbde47a077a/volumes"
Sep 16 13:04:36.727695 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:36.727659    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"]
Sep 16 13:04:36.728126 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:36.727857    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs" podUID="462e0727-223b-4217-af33-aae07198ca07" containerName="istio-proxy" containerID="cri-o://72aa2e5afdc2a11695fe68dc2b8000ba423faccc22453e7adc282ff543039066" gracePeriod=30
Sep 16 13:04:39.046892 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:39.046854    2577 patch_prober.go:28] interesting pod/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 13:04:39.047278 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:39.046911    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs" podUID="462e0727-223b-4217-af33-aae07198ca07" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:04:41.986486 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:41.986463    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"
Sep 16 13:04:42.110815 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:42.110785    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/462e0727-223b-4217-af33-aae07198ca07-istio-podinfo\") pod \"462e0727-223b-4217-af33-aae07198ca07\" (UID: \"462e0727-223b-4217-af33-aae07198ca07\") "
Sep 16 13:04:42.110954 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:42.110826    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/462e0727-223b-4217-af33-aae07198ca07-istio-envoy\") pod \"462e0727-223b-4217-af33-aae07198ca07\" (UID: \"462e0727-223b-4217-af33-aae07198ca07\") "
Sep 16 13:04:42.110954 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:42.110859    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/462e0727-223b-4217-af33-aae07198ca07-istio-data\") pod \"462e0727-223b-4217-af33-aae07198ca07\" (UID: \"462e0727-223b-4217-af33-aae07198ca07\") "
Sep 16 13:04:42.110954 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:42.110889    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/462e0727-223b-4217-af33-aae07198ca07-workload-socket\") pod \"462e0727-223b-4217-af33-aae07198ca07\" (UID: \"462e0727-223b-4217-af33-aae07198ca07\") "
Sep 16 13:04:42.110954 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:42.110904    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/462e0727-223b-4217-af33-aae07198ca07-credential-socket\") pod \"462e0727-223b-4217-af33-aae07198ca07\" (UID: \"462e0727-223b-4217-af33-aae07198ca07\") "
Sep 16 13:04:42.110954 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:42.110924    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/462e0727-223b-4217-af33-aae07198ca07-istiod-ca-cert\") pod \"462e0727-223b-4217-af33-aae07198ca07\" (UID: \"462e0727-223b-4217-af33-aae07198ca07\") "
Sep 16 13:04:42.111251 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:42.110977    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/462e0727-223b-4217-af33-aae07198ca07-workload-certs\") pod \"462e0727-223b-4217-af33-aae07198ca07\" (UID: \"462e0727-223b-4217-af33-aae07198ca07\") "
Sep 16 13:04:42.111251 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:42.111060    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfwr7\" (UniqueName: \"kubernetes.io/projected/462e0727-223b-4217-af33-aae07198ca07-kube-api-access-zfwr7\") pod \"462e0727-223b-4217-af33-aae07198ca07\" (UID: \"462e0727-223b-4217-af33-aae07198ca07\") "
Sep 16 13:04:42.111251 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:42.111144    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/462e0727-223b-4217-af33-aae07198ca07-istio-token\") pod \"462e0727-223b-4217-af33-aae07198ca07\" (UID: \"462e0727-223b-4217-af33-aae07198ca07\") "
Sep 16 13:04:42.111251 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:42.111207    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462e0727-223b-4217-af33-aae07198ca07-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "462e0727-223b-4217-af33-aae07198ca07" (UID: "462e0727-223b-4217-af33-aae07198ca07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:04:42.111251 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:42.111222    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462e0727-223b-4217-af33-aae07198ca07-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "462e0727-223b-4217-af33-aae07198ca07" (UID: "462e0727-223b-4217-af33-aae07198ca07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:04:42.111445 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:42.111233    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462e0727-223b-4217-af33-aae07198ca07-istio-data" (OuterVolumeSpecName: "istio-data") pod "462e0727-223b-4217-af33-aae07198ca07" (UID: "462e0727-223b-4217-af33-aae07198ca07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:04:42.111445 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:42.111379    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462e0727-223b-4217-af33-aae07198ca07-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "462e0727-223b-4217-af33-aae07198ca07" (UID: "462e0727-223b-4217-af33-aae07198ca07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:04:42.111445 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:42.111404    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/462e0727-223b-4217-af33-aae07198ca07-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "462e0727-223b-4217-af33-aae07198ca07" (UID: "462e0727-223b-4217-af33-aae07198ca07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:04:42.111583 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:42.111552    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/462e0727-223b-4217-af33-aae07198ca07-istio-data\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:04:42.111583 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:42.111570    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/462e0727-223b-4217-af33-aae07198ca07-workload-socket\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:04:42.111691 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:42.111585    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/462e0727-223b-4217-af33-aae07198ca07-credential-socket\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:04:42.111691 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:42.111599    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/462e0727-223b-4217-af33-aae07198ca07-istiod-ca-cert\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:04:42.111691 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:42.111611    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/462e0727-223b-4217-af33-aae07198ca07-workload-certs\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:04:42.113417 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:42.113386    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/462e0727-223b-4217-af33-aae07198ca07-kube-api-access-zfwr7" (OuterVolumeSpecName: "kube-api-access-zfwr7") pod "462e0727-223b-4217-af33-aae07198ca07" (UID: "462e0727-223b-4217-af33-aae07198ca07"). InnerVolumeSpecName "kube-api-access-zfwr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:04:42.113545 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:42.113492    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/462e0727-223b-4217-af33-aae07198ca07-istio-token" (OuterVolumeSpecName: "istio-token") pod "462e0727-223b-4217-af33-aae07198ca07" (UID: "462e0727-223b-4217-af33-aae07198ca07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:04:42.113545 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:42.113494    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/462e0727-223b-4217-af33-aae07198ca07-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "462e0727-223b-4217-af33-aae07198ca07" (UID: "462e0727-223b-4217-af33-aae07198ca07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 13:04:42.113795 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:42.113779    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462e0727-223b-4217-af33-aae07198ca07-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "462e0727-223b-4217-af33-aae07198ca07" (UID: "462e0727-223b-4217-af33-aae07198ca07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:04:42.212532 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:42.212499    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfwr7\" (UniqueName: \"kubernetes.io/projected/462e0727-223b-4217-af33-aae07198ca07-kube-api-access-zfwr7\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:04:42.212532 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:42.212528    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/462e0727-223b-4217-af33-aae07198ca07-istio-token\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:04:42.212532 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:42.212538    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/462e0727-223b-4217-af33-aae07198ca07-istio-podinfo\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:04:42.212760 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:42.212546    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/462e0727-223b-4217-af33-aae07198ca07-istio-envoy\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:04:42.678496 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:42.678464    2577 generic.go:356] "Generic (PLEG): container finished" podID="462e0727-223b-4217-af33-aae07198ca07" containerID="72aa2e5afdc2a11695fe68dc2b8000ba423faccc22453e7adc282ff543039066" exitCode=0
Sep 16 13:04:42.678676 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:42.678531    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"
Sep 16 13:04:42.678676 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:42.678545    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs" event={"ID":"462e0727-223b-4217-af33-aae07198ca07","Type":"ContainerDied","Data":"72aa2e5afdc2a11695fe68dc2b8000ba423faccc22453e7adc282ff543039066"}
Sep 16 13:04:42.678676 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:42.678635    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs" event={"ID":"462e0727-223b-4217-af33-aae07198ca07","Type":"ContainerDied","Data":"1c95c22c973d679355f93354ba45c15164ad7a07856b96e424e1e9065a4e2395"}
Sep 16 13:04:42.678676 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:42.678665    2577 scope.go:117] "RemoveContainer" containerID="72aa2e5afdc2a11695fe68dc2b8000ba423faccc22453e7adc282ff543039066"
Sep 16 13:04:42.689754 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:42.689720    2577 scope.go:117] "RemoveContainer" containerID="c8f872f61ddab2e36fd291728ca8cf8f7faa954725d521c6dc6c328eb7775045"
Sep 16 13:04:42.699095 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:42.699068    2577 scope.go:117] "RemoveContainer" containerID="72aa2e5afdc2a11695fe68dc2b8000ba423faccc22453e7adc282ff543039066"
Sep 16 13:04:42.699719 ip-10-0-133-190 kubenswrapper[2577]: E0916 13:04:42.699425    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72aa2e5afdc2a11695fe68dc2b8000ba423faccc22453e7adc282ff543039066\": container with ID starting with 72aa2e5afdc2a11695fe68dc2b8000ba423faccc22453e7adc282ff543039066 not found: ID does not exist" containerID="72aa2e5afdc2a11695fe68dc2b8000ba423faccc22453e7adc282ff543039066"
Sep 16 13:04:42.699719 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:42.699463    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72aa2e5afdc2a11695fe68dc2b8000ba423faccc22453e7adc282ff543039066"} err="failed to get container status \"72aa2e5afdc2a11695fe68dc2b8000ba423faccc22453e7adc282ff543039066\": rpc error: code = NotFound desc = could not find container \"72aa2e5afdc2a11695fe68dc2b8000ba423faccc22453e7adc282ff543039066\": container with ID starting with 72aa2e5afdc2a11695fe68dc2b8000ba423faccc22453e7adc282ff543039066 not found: ID does not exist"
Sep 16 13:04:42.699719 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:42.699486    2577 scope.go:117] "RemoveContainer" containerID="c8f872f61ddab2e36fd291728ca8cf8f7faa954725d521c6dc6c328eb7775045"
Sep 16 13:04:42.700093 ip-10-0-133-190 kubenswrapper[2577]: E0916 13:04:42.700072    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8f872f61ddab2e36fd291728ca8cf8f7faa954725d521c6dc6c328eb7775045\": container with ID starting with c8f872f61ddab2e36fd291728ca8cf8f7faa954725d521c6dc6c328eb7775045 not found: ID does not exist" containerID="c8f872f61ddab2e36fd291728ca8cf8f7faa954725d521c6dc6c328eb7775045"
Sep 16 13:04:42.700242 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:42.700212    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8f872f61ddab2e36fd291728ca8cf8f7faa954725d521c6dc6c328eb7775045"} err="failed to get container status \"c8f872f61ddab2e36fd291728ca8cf8f7faa954725d521c6dc6c328eb7775045\": rpc error: code = NotFound desc = could not find container \"c8f872f61ddab2e36fd291728ca8cf8f7faa954725d521c6dc6c328eb7775045\": container with ID starting with c8f872f61ddab2e36fd291728ca8cf8f7faa954725d521c6dc6c328eb7775045 not found: ID does not exist"
Sep 16 13:04:42.702258 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:42.702239    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"]
Sep 16 13:04:42.706619 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:42.706600    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-490797-openshift-default-7648768bbd-rcsxs"]
Sep 16 13:04:44.055963 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:04:44.055929    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="462e0727-223b-4217-af33-aae07198ca07" path="/var/lib/kubelet/pods/462e0727-223b-4217-af33-aae07198ca07/volumes"
Sep 16 13:05:07.883108 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:07.883017    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-524adf28-p9pfc"]
Sep 16 13:05:07.883553 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:07.883479    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb83e365-cc67-44bc-9a58-f1eb36ff6232" containerName="payload-pre-processing"
Sep 16 13:05:07.883553 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:07.883490    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb83e365-cc67-44bc-9a58-f1eb36ff6232" containerName="payload-pre-processing"
Sep 16 13:05:07.883553 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:07.883500    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9742c8d6-9162-460b-8426-ecbde47a077a" containerName="main"
Sep 16 13:05:07.883553 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:07.883505    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9742c8d6-9162-460b-8426-ecbde47a077a" containerName="main"
Sep 16 13:05:07.883553 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:07.883522    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95d9c301-cdb6-4356-9fe1-fb91a8da0c83" containerName="main"
Sep 16 13:05:07.883553 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:07.883527    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="95d9c301-cdb6-4356-9fe1-fb91a8da0c83" containerName="main"
Sep 16 13:05:07.883553 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:07.883534    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34141f56-6c73-4ac9-adc6-0da953fca281" containerName="main"
Sep 16 13:05:07.883553 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:07.883539    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="34141f56-6c73-4ac9-adc6-0da953fca281" containerName="main"
Sep 16 13:05:07.883553 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:07.883550    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9741c7d0-a898-4b78-a0a6-e32288c5a78b" containerName="revoke-keys"
Sep 16 13:05:07.883553 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:07.883556    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9741c7d0-a898-4b78-a0a6-e32288c5a78b" containerName="revoke-keys"
Sep 16 13:05:07.883915 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:07.883564    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="462e0727-223b-4217-af33-aae07198ca07" containerName="istio-proxy"
Sep 16 13:05:07.883915 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:07.883569    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="462e0727-223b-4217-af33-aae07198ca07" containerName="istio-proxy"
Sep 16 13:05:07.883915 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:07.883575    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="462e0727-223b-4217-af33-aae07198ca07" containerName="istio-proxy"
Sep 16 13:05:07.883915 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:07.883581    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="462e0727-223b-4217-af33-aae07198ca07" containerName="istio-proxy"
Sep 16 13:05:07.887197 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:07.884348    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="95d9c301-cdb6-4356-9fe1-fb91a8da0c83" containerName="main"
Sep 16 13:05:07.887197 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:07.884388    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9741c7d0-a898-4b78-a0a6-e32288c5a78b" containerName="revoke-keys"
Sep 16 13:05:07.887197 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:07.884411    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="462e0727-223b-4217-af33-aae07198ca07" containerName="istio-proxy"
Sep 16 13:05:07.887197 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:07.884422    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="34141f56-6c73-4ac9-adc6-0da953fca281" containerName="main"
Sep 16 13:05:07.887197 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:07.884440    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb83e365-cc67-44bc-9a58-f1eb36ff6232" containerName="payload-pre-processing"
Sep 16 13:05:07.887197 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:07.884459    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9742c8d6-9162-460b-8426-ecbde47a077a" containerName="main"
Sep 16 13:05:07.891887 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:07.891658    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-524adf28-p9pfc"
Sep 16 13:05:07.895050 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:07.895002    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-64zph\""
Sep 16 13:05:07.895712 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:07.895385    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 13:05:07.898149 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:07.898123    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-524adf28-p9pfc"]
Sep 16 13:05:07.970624 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:07.970592    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42551886-90ee-4c90-94bc-3b15d8399253-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-524adf28-p9pfc\" (UID: \"42551886-90ee-4c90-94bc-3b15d8399253\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-524adf28-p9pfc"
Sep 16 13:05:08.071827 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:08.071787    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42551886-90ee-4c90-94bc-3b15d8399253-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-524adf28-p9pfc\" (UID: \"42551886-90ee-4c90-94bc-3b15d8399253\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-524adf28-p9pfc"
Sep 16 13:05:08.072401 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:08.072382    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42551886-90ee-4c90-94bc-3b15d8399253-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-524adf28-p9pfc\" (UID: \"42551886-90ee-4c90-94bc-3b15d8399253\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-524adf28-p9pfc"
Sep 16 13:05:08.207230 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:08.207149    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-524adf28-p9pfc"
Sep 16 13:05:08.484966 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:08.484938    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-524adf28-p9pfc"]
Sep 16 13:05:08.486911 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:05:08.486880    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42551886_90ee_4c90_94bc_3b15d8399253.slice/crio-0afbb7aace0906d9967c09f288a882d3d15334937ff8b8ff73e7ceb67ca9b348 WatchSource:0}: Error finding container 0afbb7aace0906d9967c09f288a882d3d15334937ff8b8ff73e7ceb67ca9b348: Status 404 returned error can't find the container with id 0afbb7aace0906d9967c09f288a882d3d15334937ff8b8ff73e7ceb67ca9b348
Sep 16 13:05:08.846246 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:08.846211    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-524adf28-p9pfc" event={"ID":"42551886-90ee-4c90-94bc-3b15d8399253","Type":"ContainerStarted","Data":"aa147a812b8491206e69829eedfeae2ff45c5bc43ca651a446bfc8d58f1349ad"}
Sep 16 13:05:08.846418 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:08.846263    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-524adf28-p9pfc" event={"ID":"42551886-90ee-4c90-94bc-3b15d8399253","Type":"ContainerStarted","Data":"0afbb7aace0906d9967c09f288a882d3d15334937ff8b8ff73e7ceb67ca9b348"}
Sep 16 13:05:08.863811 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:08.863764    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-524adf28-p9pfc" podStartSLOduration=1.863749243 podStartE2EDuration="1.863749243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:05:08.861897045 +0000 UTC m=+1615.309129687" watchObservedRunningTime="2026-09-16 13:05:08.863749243 +0000 UTC m=+1615.310981963"
Sep 16 13:05:09.854361 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:09.854325    2577 generic.go:356] "Generic (PLEG): container finished" podID="42551886-90ee-4c90-94bc-3b15d8399253" containerID="aa147a812b8491206e69829eedfeae2ff45c5bc43ca651a446bfc8d58f1349ad" exitCode=0
Sep 16 13:05:09.854809 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:09.854416    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-524adf28-p9pfc" event={"ID":"42551886-90ee-4c90-94bc-3b15d8399253","Type":"ContainerDied","Data":"aa147a812b8491206e69829eedfeae2ff45c5bc43ca651a446bfc8d58f1349ad"}
Sep 16 13:05:11.013285 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:11.013260    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-524adf28-p9pfc"
Sep 16 13:05:11.102806 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:11.102773    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42551886-90ee-4c90-94bc-3b15d8399253-maas-api-service-ca\") pod \"42551886-90ee-4c90-94bc-3b15d8399253\" (UID: \"42551886-90ee-4c90-94bc-3b15d8399253\") "
Sep 16 13:05:11.103143 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:11.103122    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42551886-90ee-4c90-94bc-3b15d8399253-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "42551886-90ee-4c90-94bc-3b15d8399253" (UID: "42551886-90ee-4c90-94bc-3b15d8399253"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:05:11.203991 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:11.203897    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42551886-90ee-4c90-94bc-3b15d8399253-maas-api-service-ca\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:05:11.869387 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:11.869347    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-524adf28-p9pfc" event={"ID":"42551886-90ee-4c90-94bc-3b15d8399253","Type":"ContainerDied","Data":"0afbb7aace0906d9967c09f288a882d3d15334937ff8b8ff73e7ceb67ca9b348"}
Sep 16 13:05:11.869551 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:11.869407    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0afbb7aace0906d9967c09f288a882d3d15334937ff8b8ff73e7ceb67ca9b348"
Sep 16 13:05:11.869551 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:11.869359    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-524adf28-p9pfc"
Sep 16 13:05:15.358791 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:15.358754    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7d0256/distinct-w4-7d0256-kserve-556b55b77-xcvvp"]
Sep 16 13:05:15.359315 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:15.359297    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42551886-90ee-4c90-94bc-3b15d8399253" containerName="revoke-keys"
Sep 16 13:05:15.359384 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:15.359318    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="42551886-90ee-4c90-94bc-3b15d8399253" containerName="revoke-keys"
Sep 16 13:05:15.359450 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:15.359405    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="42551886-90ee-4c90-94bc-3b15d8399253" containerName="revoke-keys"
Sep 16 13:05:15.359450 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:15.359414    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="462e0727-223b-4217-af33-aae07198ca07" containerName="istio-proxy"
Sep 16 13:05:15.363880 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:15.363859    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d0256/distinct-w4-7d0256-kserve-556b55b77-xcvvp"
Sep 16 13:05:15.366553 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:15.366531    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7d0256\"/\"distinct-w4-7d0256-kserve-self-signed-certs\""
Sep 16 13:05:15.371402 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:15.371377    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7d0256/distinct-w4-7d0256-kserve-556b55b77-xcvvp"]
Sep 16 13:05:15.450370 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:15.450340    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7735aca7-7d6b-464d-872a-00a4709feaec-dshm\") pod \"distinct-w4-7d0256-kserve-556b55b77-xcvvp\" (UID: \"7735aca7-7d6b-464d-872a-00a4709feaec\") " pod="e2e-models-e2e-worker-w4-7d0256/distinct-w4-7d0256-kserve-556b55b77-xcvvp"
Sep 16 13:05:15.450540 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:15.450414    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7735aca7-7d6b-464d-872a-00a4709feaec-model-cache\") pod \"distinct-w4-7d0256-kserve-556b55b77-xcvvp\" (UID: \"7735aca7-7d6b-464d-872a-00a4709feaec\") " pod="e2e-models-e2e-worker-w4-7d0256/distinct-w4-7d0256-kserve-556b55b77-xcvvp"
Sep 16 13:05:15.450540 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:15.450523    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7735aca7-7d6b-464d-872a-00a4709feaec-tmp-dir\") pod \"distinct-w4-7d0256-kserve-556b55b77-xcvvp\" (UID: \"7735aca7-7d6b-464d-872a-00a4709feaec\") " pod="e2e-models-e2e-worker-w4-7d0256/distinct-w4-7d0256-kserve-556b55b77-xcvvp"
Sep 16 13:05:15.450636 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:15.450559    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7735aca7-7d6b-464d-872a-00a4709feaec-home\") pod \"distinct-w4-7d0256-kserve-556b55b77-xcvvp\" (UID: \"7735aca7-7d6b-464d-872a-00a4709feaec\") " pod="e2e-models-e2e-worker-w4-7d0256/distinct-w4-7d0256-kserve-556b55b77-xcvvp"
Sep 16 13:05:15.450636 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:15.450601    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwkw6\" (UniqueName: \"kubernetes.io/projected/7735aca7-7d6b-464d-872a-00a4709feaec-kube-api-access-cwkw6\") pod \"distinct-w4-7d0256-kserve-556b55b77-xcvvp\" (UID: \"7735aca7-7d6b-464d-872a-00a4709feaec\") " pod="e2e-models-e2e-worker-w4-7d0256/distinct-w4-7d0256-kserve-556b55b77-xcvvp"
Sep 16 13:05:15.450636 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:15.450631    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7735aca7-7d6b-464d-872a-00a4709feaec-tls-certs\") pod \"distinct-w4-7d0256-kserve-556b55b77-xcvvp\" (UID: \"7735aca7-7d6b-464d-872a-00a4709feaec\") " pod="e2e-models-e2e-worker-w4-7d0256/distinct-w4-7d0256-kserve-556b55b77-xcvvp"
Sep 16 13:05:15.552235 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:15.552199    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7735aca7-7d6b-464d-872a-00a4709feaec-tmp-dir\") pod \"distinct-w4-7d0256-kserve-556b55b77-xcvvp\" (UID: \"7735aca7-7d6b-464d-872a-00a4709feaec\") " pod="e2e-models-e2e-worker-w4-7d0256/distinct-w4-7d0256-kserve-556b55b77-xcvvp"
Sep 16 13:05:15.552235 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:15.552240    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7735aca7-7d6b-464d-872a-00a4709feaec-home\") pod \"distinct-w4-7d0256-kserve-556b55b77-xcvvp\" (UID: \"7735aca7-7d6b-464d-872a-00a4709feaec\") " pod="e2e-models-e2e-worker-w4-7d0256/distinct-w4-7d0256-kserve-556b55b77-xcvvp"
Sep 16 13:05:15.552467 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:15.552366    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwkw6\" (UniqueName: \"kubernetes.io/projected/7735aca7-7d6b-464d-872a-00a4709feaec-kube-api-access-cwkw6\") pod \"distinct-w4-7d0256-kserve-556b55b77-xcvvp\" (UID: \"7735aca7-7d6b-464d-872a-00a4709feaec\") " pod="e2e-models-e2e-worker-w4-7d0256/distinct-w4-7d0256-kserve-556b55b77-xcvvp"
Sep 16 13:05:15.552467 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:15.552410    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7735aca7-7d6b-464d-872a-00a4709feaec-tls-certs\") pod \"distinct-w4-7d0256-kserve-556b55b77-xcvvp\" (UID: \"7735aca7-7d6b-464d-872a-00a4709feaec\") " pod="e2e-models-e2e-worker-w4-7d0256/distinct-w4-7d0256-kserve-556b55b77-xcvvp"
Sep 16 13:05:15.552570 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:15.552486    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7735aca7-7d6b-464d-872a-00a4709feaec-dshm\") pod \"distinct-w4-7d0256-kserve-556b55b77-xcvvp\" (UID: \"7735aca7-7d6b-464d-872a-00a4709feaec\") " pod="e2e-models-e2e-worker-w4-7d0256/distinct-w4-7d0256-kserve-556b55b77-xcvvp"
Sep 16 13:05:15.552570 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:15.552543    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7735aca7-7d6b-464d-872a-00a4709feaec-model-cache\") pod \"distinct-w4-7d0256-kserve-556b55b77-xcvvp\" (UID: \"7735aca7-7d6b-464d-872a-00a4709feaec\") " pod="e2e-models-e2e-worker-w4-7d0256/distinct-w4-7d0256-kserve-556b55b77-xcvvp"
Sep 16 13:05:15.552681 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:15.552575    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7735aca7-7d6b-464d-872a-00a4709feaec-home\") pod \"distinct-w4-7d0256-kserve-556b55b77-xcvvp\" (UID: \"7735aca7-7d6b-464d-872a-00a4709feaec\") " pod="e2e-models-e2e-worker-w4-7d0256/distinct-w4-7d0256-kserve-556b55b77-xcvvp"
Sep 16 13:05:15.552681 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:15.552631    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7735aca7-7d6b-464d-872a-00a4709feaec-tmp-dir\") pod \"distinct-w4-7d0256-kserve-556b55b77-xcvvp\" (UID: \"7735aca7-7d6b-464d-872a-00a4709feaec\") " pod="e2e-models-e2e-worker-w4-7d0256/distinct-w4-7d0256-kserve-556b55b77-xcvvp"
Sep 16 13:05:15.552850 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:15.552832    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7735aca7-7d6b-464d-872a-00a4709feaec-model-cache\") pod \"distinct-w4-7d0256-kserve-556b55b77-xcvvp\" (UID: \"7735aca7-7d6b-464d-872a-00a4709feaec\") " pod="e2e-models-e2e-worker-w4-7d0256/distinct-w4-7d0256-kserve-556b55b77-xcvvp"
Sep 16 13:05:15.554848 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:15.554826    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7735aca7-7d6b-464d-872a-00a4709feaec-dshm\") pod \"distinct-w4-7d0256-kserve-556b55b77-xcvvp\" (UID: \"7735aca7-7d6b-464d-872a-00a4709feaec\") " pod="e2e-models-e2e-worker-w4-7d0256/distinct-w4-7d0256-kserve-556b55b77-xcvvp"
Sep 16 13:05:15.555042 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:15.555008    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7735aca7-7d6b-464d-872a-00a4709feaec-tls-certs\") pod \"distinct-w4-7d0256-kserve-556b55b77-xcvvp\" (UID: \"7735aca7-7d6b-464d-872a-00a4709feaec\") " pod="e2e-models-e2e-worker-w4-7d0256/distinct-w4-7d0256-kserve-556b55b77-xcvvp"
Sep 16 13:05:15.561506 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:15.561474    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwkw6\" (UniqueName: \"kubernetes.io/projected/7735aca7-7d6b-464d-872a-00a4709feaec-kube-api-access-cwkw6\") pod \"distinct-w4-7d0256-kserve-556b55b77-xcvvp\" (UID: \"7735aca7-7d6b-464d-872a-00a4709feaec\") " pod="e2e-models-e2e-worker-w4-7d0256/distinct-w4-7d0256-kserve-556b55b77-xcvvp"
Sep 16 13:05:15.676813 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:15.676721    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d0256/distinct-w4-7d0256-kserve-556b55b77-xcvvp"
Sep 16 13:05:15.954601 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:15.954569    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7d0256/distinct-w4-7d0256-kserve-556b55b77-xcvvp"]
Sep 16 13:05:15.955639 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:05:15.955612    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7735aca7_7d6b_464d_872a_00a4709feaec.slice/crio-0ff3b33e27259e7ccc8b41cf5b35bfa19d946cb9db0a03259b6f6d94234ef208 WatchSource:0}: Error finding container 0ff3b33e27259e7ccc8b41cf5b35bfa19d946cb9db0a03259b6f6d94234ef208: Status 404 returned error can't find the container with id 0ff3b33e27259e7ccc8b41cf5b35bfa19d946cb9db0a03259b6f6d94234ef208
Sep 16 13:05:16.903467 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:16.903432    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d0256/distinct-w4-7d0256-kserve-556b55b77-xcvvp" event={"ID":"7735aca7-7d6b-464d-872a-00a4709feaec","Type":"ContainerStarted","Data":"0abc2760f53a02526e172e88dbf7d1372f0486058d9830cd51ce2cef97a8955c"}
Sep 16 13:05:16.903467 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:16.903474    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d0256/distinct-w4-7d0256-kserve-556b55b77-xcvvp" event={"ID":"7735aca7-7d6b-464d-872a-00a4709feaec","Type":"ContainerStarted","Data":"0ff3b33e27259e7ccc8b41cf5b35bfa19d946cb9db0a03259b6f6d94234ef208"}
Sep 16 13:05:16.921816 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:16.921770    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7d0256/distinct-w4-7d0256-kserve-556b55b77-xcvvp" podStartSLOduration=1.921756268 podStartE2EDuration="1.921756268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:05:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:05:16.921080485 +0000 UTC m=+1623.368313128" watchObservedRunningTime="2026-09-16 13:05:16.921756268 +0000 UTC m=+1623.368988902"
Sep 16 13:05:23.604339 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:23.604308    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-524adf28-79c5d568d6-cc4ws"]
Sep 16 13:05:23.604871 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:23.604563    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-524adf28-79c5d568d6-cc4ws" podUID="cc23485f-533d-44aa-82be-d16f577d09ec" containerName="payload-pre-processing" containerID="cri-o://3a8831952bf67e681d03a92ccb30f7ee56c050a8efd93c0b83b55d9643f99ffd" gracePeriod=30
Sep 16 13:05:23.809201 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:23.809171    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-524adf28-p9pfc"]
Sep 16 13:05:23.814407 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:23.814382    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-524adf28-p9pfc"]
Sep 16 13:05:23.865768 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:23.865704    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-524adf28-79c5d568d6-cc4ws"
Sep 16 13:05:23.954605 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:23.954574    2577 generic.go:356] "Generic (PLEG): container finished" podID="cc23485f-533d-44aa-82be-d16f577d09ec" containerID="3a8831952bf67e681d03a92ccb30f7ee56c050a8efd93c0b83b55d9643f99ffd" exitCode=0
Sep 16 13:05:23.954788 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:23.954625    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-524adf28-79c5d568d6-cc4ws"
Sep 16 13:05:23.954788 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:23.954631    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-524adf28-79c5d568d6-cc4ws" event={"ID":"cc23485f-533d-44aa-82be-d16f577d09ec","Type":"ContainerDied","Data":"3a8831952bf67e681d03a92ccb30f7ee56c050a8efd93c0b83b55d9643f99ffd"}
Sep 16 13:05:23.954788 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:23.954674    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-524adf28-79c5d568d6-cc4ws" event={"ID":"cc23485f-533d-44aa-82be-d16f577d09ec","Type":"ContainerDied","Data":"71fb96d44d69fa849cb14ec97ac9e19e365cfbe2f43c829046c82994b0380170"}
Sep 16 13:05:23.954788 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:23.954696    2577 scope.go:117] "RemoveContainer" containerID="3a8831952bf67e681d03a92ccb30f7ee56c050a8efd93c0b83b55d9643f99ffd"
Sep 16 13:05:23.964436 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:23.964412    2577 scope.go:117] "RemoveContainer" containerID="3a8831952bf67e681d03a92ccb30f7ee56c050a8efd93c0b83b55d9643f99ffd"
Sep 16 13:05:23.964713 ip-10-0-133-190 kubenswrapper[2577]: E0916 13:05:23.964695    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a8831952bf67e681d03a92ccb30f7ee56c050a8efd93c0b83b55d9643f99ffd\": container with ID starting with 3a8831952bf67e681d03a92ccb30f7ee56c050a8efd93c0b83b55d9643f99ffd not found: ID does not exist" containerID="3a8831952bf67e681d03a92ccb30f7ee56c050a8efd93c0b83b55d9643f99ffd"
Sep 16 13:05:23.964781 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:23.964723    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a8831952bf67e681d03a92ccb30f7ee56c050a8efd93c0b83b55d9643f99ffd"} err="failed to get container status \"3a8831952bf67e681d03a92ccb30f7ee56c050a8efd93c0b83b55d9643f99ffd\": rpc error: code = NotFound desc = could not find container \"3a8831952bf67e681d03a92ccb30f7ee56c050a8efd93c0b83b55d9643f99ffd\": container with ID starting with 3a8831952bf67e681d03a92ccb30f7ee56c050a8efd93c0b83b55d9643f99ffd not found: ID does not exist"
Sep 16 13:05:24.046105 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:24.046074    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc23485f-533d-44aa-82be-d16f577d09ec-plugins-config-volume\") pod \"cc23485f-533d-44aa-82be-d16f577d09ec\" (UID: \"cc23485f-533d-44aa-82be-d16f577d09ec\") "
Sep 16 13:05:24.046268 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:24.046189    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bw5xd\" (UniqueName: \"kubernetes.io/projected/cc23485f-533d-44aa-82be-d16f577d09ec-kube-api-access-bw5xd\") pod \"cc23485f-533d-44aa-82be-d16f577d09ec\" (UID: \"cc23485f-533d-44aa-82be-d16f577d09ec\") "
Sep 16 13:05:24.046453 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:24.046430    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc23485f-533d-44aa-82be-d16f577d09ec-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cc23485f-533d-44aa-82be-d16f577d09ec" (UID: "cc23485f-533d-44aa-82be-d16f577d09ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:05:24.048541 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:24.048507    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc23485f-533d-44aa-82be-d16f577d09ec-kube-api-access-bw5xd" (OuterVolumeSpecName: "kube-api-access-bw5xd") pod "cc23485f-533d-44aa-82be-d16f577d09ec" (UID: "cc23485f-533d-44aa-82be-d16f577d09ec"). InnerVolumeSpecName "kube-api-access-bw5xd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:05:24.056920 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:24.056893    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42551886-90ee-4c90-94bc-3b15d8399253" path="/var/lib/kubelet/pods/42551886-90ee-4c90-94bc-3b15d8399253/volumes"
Sep 16 13:05:24.147390 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:24.147309    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw5xd\" (UniqueName: \"kubernetes.io/projected/cc23485f-533d-44aa-82be-d16f577d09ec-kube-api-access-bw5xd\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:05:24.147390 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:24.147337    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc23485f-533d-44aa-82be-d16f577d09ec-plugins-config-volume\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:05:24.270846 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:24.270815    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-524adf28-79c5d568d6-cc4ws"]
Sep 16 13:05:24.277508 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:24.277478    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-524adf28-79c5d568d6-cc4ws"]
Sep 16 13:05:25.677480 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:25.677447    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7d0256/distinct-w4-7d0256-kserve-556b55b77-xcvvp"
Sep 16 13:05:25.677861 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:25.677511    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7d0256/distinct-w4-7d0256-kserve-556b55b77-xcvvp"
Sep 16 13:05:25.691202 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:25.691173    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7d0256/distinct-w4-7d0256-kserve-556b55b77-xcvvp"
Sep 16 13:05:25.980007 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:25.979924    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7d0256/distinct-w4-7d0256-kserve-556b55b77-xcvvp"
Sep 16 13:05:26.055580 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:05:26.055549    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc23485f-533d-44aa-82be-d16f577d09ec" path="/var/lib/kubelet/pods/cc23485f-533d-44aa-82be-d16f577d09ec/volumes"
Sep 16 13:06:14.875049 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:06:14.870992    2577 scope.go:117] "RemoveContainer" containerID="76266e1a952d9de5860fe275ceb935f453e9917a292286755748dca3fab9892c"
Sep 16 13:06:14.886590 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:06:14.886565    2577 scope.go:117] "RemoveContainer" containerID="f09b85ff6a528b90f92fa7735a36d89dbe2f71e1aba8fd63f82c928f8d0014e3"
Sep 16 13:06:14.896584 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:06:14.896563    2577 scope.go:117] "RemoveContainer" containerID="49189360a08255f884c6281358e304cf716d3e3b01bc27ebbc18e19aa1bf59b2"
Sep 16 13:06:14.906861 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:06:14.906842    2577 scope.go:117] "RemoveContainer" containerID="09bd2338ea8f03606683a5028089867bf7daf6314a08ea163f01f3e953fec78f"
Sep 16 13:07:14.946640 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:14.946611    2577 scope.go:117] "RemoveContainer" containerID="beb483b9eb16af52efc711f5bd89b9823d296264c1565cb61190cf931858a6a4"
Sep 16 13:07:14.956099 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:14.956078    2577 scope.go:117] "RemoveContainer" containerID="069b37734d2e1aef01e6925cc7d70d4ef0fa8f9274e04d2724d038c2ea40c7a1"
Sep 16 13:07:17.846844 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:17.846797    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f0722f-57df84c966-jqqj2"]
Sep 16 13:07:17.847294 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:17.847281    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc23485f-533d-44aa-82be-d16f577d09ec" containerName="payload-pre-processing"
Sep 16 13:07:17.847294 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:17.847295    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc23485f-533d-44aa-82be-d16f577d09ec" containerName="payload-pre-processing"
Sep 16 13:07:17.847371 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:17.847359    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc23485f-533d-44aa-82be-d16f577d09ec" containerName="payload-pre-processing"
Sep 16 13:07:17.850542 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:17.850528    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0722f-57df84c966-jqqj2"
Sep 16 13:07:17.853617 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:17.853588    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f0722f-dockercfg-ww64b\""
Sep 16 13:07:17.853617 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:17.853608    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f0722f\""
Sep 16 13:07:17.858824 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:17.858802    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f0722f-57df84c966-jqqj2"]
Sep 16 13:07:17.874664 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:17.874632    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f0722f-57b85b59f7-2j54q"]
Sep 16 13:07:17.878944 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:17.878924    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f0722f-57b85b59f7-2j54q"
Sep 16 13:07:17.888868 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:17.888847    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f0722f-57b85b59f7-2j54q"]
Sep 16 13:07:18.002292 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:18.002256    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e1cd797-8cf2-46ec-8779-30f3fa51d5da-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f0722f-57df84c966-jqqj2\" (UID: \"4e1cd797-8cf2-46ec-8779-30f3fa51d5da\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0722f-57df84c966-jqqj2"
Sep 16 13:07:18.002454 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:18.002321    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h87tz\" (UniqueName: \"kubernetes.io/projected/2389a693-9d04-4a93-97df-0649e5f7941e-kube-api-access-h87tz\") pod \"payload-processing-e2e-api-a-f0722f-57b85b59f7-2j54q\" (UID: \"2389a693-9d04-4a93-97df-0649e5f7941e\") " pod="openshift-ingress/payload-processing-e2e-api-a-f0722f-57b85b59f7-2j54q"
Sep 16 13:07:18.002454 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:18.002343    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znp66\" (UniqueName: \"kubernetes.io/projected/4e1cd797-8cf2-46ec-8779-30f3fa51d5da-kube-api-access-znp66\") pod \"payload-pre-processing-e2e-api-a-f0722f-57df84c966-jqqj2\" (UID: \"4e1cd797-8cf2-46ec-8779-30f3fa51d5da\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0722f-57df84c966-jqqj2"
Sep 16 13:07:18.002454 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:18.002363    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2389a693-9d04-4a93-97df-0649e5f7941e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f0722f-57b85b59f7-2j54q\" (UID: \"2389a693-9d04-4a93-97df-0649e5f7941e\") " pod="openshift-ingress/payload-processing-e2e-api-a-f0722f-57b85b59f7-2j54q"
Sep 16 13:07:18.102968 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:18.102886    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h87tz\" (UniqueName: \"kubernetes.io/projected/2389a693-9d04-4a93-97df-0649e5f7941e-kube-api-access-h87tz\") pod \"payload-processing-e2e-api-a-f0722f-57b85b59f7-2j54q\" (UID: \"2389a693-9d04-4a93-97df-0649e5f7941e\") " pod="openshift-ingress/payload-processing-e2e-api-a-f0722f-57b85b59f7-2j54q"
Sep 16 13:07:18.102968 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:18.102918    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znp66\" (UniqueName: \"kubernetes.io/projected/4e1cd797-8cf2-46ec-8779-30f3fa51d5da-kube-api-access-znp66\") pod \"payload-pre-processing-e2e-api-a-f0722f-57df84c966-jqqj2\" (UID: \"4e1cd797-8cf2-46ec-8779-30f3fa51d5da\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0722f-57df84c966-jqqj2"
Sep 16 13:07:18.102968 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:18.102941    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2389a693-9d04-4a93-97df-0649e5f7941e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f0722f-57b85b59f7-2j54q\" (UID: \"2389a693-9d04-4a93-97df-0649e5f7941e\") " pod="openshift-ingress/payload-processing-e2e-api-a-f0722f-57b85b59f7-2j54q"
Sep 16 13:07:18.103296 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:18.103050    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e1cd797-8cf2-46ec-8779-30f3fa51d5da-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f0722f-57df84c966-jqqj2\" (UID: \"4e1cd797-8cf2-46ec-8779-30f3fa51d5da\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0722f-57df84c966-jqqj2"
Sep 16 13:07:18.103687 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:18.103667    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2389a693-9d04-4a93-97df-0649e5f7941e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f0722f-57b85b59f7-2j54q\" (UID: \"2389a693-9d04-4a93-97df-0649e5f7941e\") " pod="openshift-ingress/payload-processing-e2e-api-a-f0722f-57b85b59f7-2j54q"
Sep 16 13:07:18.103731 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:18.103712    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e1cd797-8cf2-46ec-8779-30f3fa51d5da-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f0722f-57df84c966-jqqj2\" (UID: \"4e1cd797-8cf2-46ec-8779-30f3fa51d5da\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0722f-57df84c966-jqqj2"
Sep 16 13:07:18.111811 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:18.111780    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h87tz\" (UniqueName: \"kubernetes.io/projected/2389a693-9d04-4a93-97df-0649e5f7941e-kube-api-access-h87tz\") pod \"payload-processing-e2e-api-a-f0722f-57b85b59f7-2j54q\" (UID: \"2389a693-9d04-4a93-97df-0649e5f7941e\") " pod="openshift-ingress/payload-processing-e2e-api-a-f0722f-57b85b59f7-2j54q"
Sep 16 13:07:18.111923 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:18.111875    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znp66\" (UniqueName: \"kubernetes.io/projected/4e1cd797-8cf2-46ec-8779-30f3fa51d5da-kube-api-access-znp66\") pod \"payload-pre-processing-e2e-api-a-f0722f-57df84c966-jqqj2\" (UID: \"4e1cd797-8cf2-46ec-8779-30f3fa51d5da\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0722f-57df84c966-jqqj2"
Sep 16 13:07:18.163862 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:18.163831    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0722f-57df84c966-jqqj2"
Sep 16 13:07:18.196204 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:18.196174    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f0722f-57b85b59f7-2j54q"
Sep 16 13:07:18.439829 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:18.439808    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f0722f-57df84c966-jqqj2"]
Sep 16 13:07:18.441546 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:07:18.441521    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e1cd797_8cf2_46ec_8779_30f3fa51d5da.slice/crio-d94f12e2d6239c613111ef10329e6c63981efed34883efff23bf1d83a7a97c04 WatchSource:0}: Error finding container d94f12e2d6239c613111ef10329e6c63981efed34883efff23bf1d83a7a97c04: Status 404 returned error can't find the container with id d94f12e2d6239c613111ef10329e6c63981efed34883efff23bf1d83a7a97c04
Sep 16 13:07:18.492157 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:18.492127    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f0722f-57b85b59f7-2j54q"]
Sep 16 13:07:18.493325 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:07:18.493296    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2389a693_9d04_4a93_97df_0649e5f7941e.slice/crio-019d9353f1ff97e533ebc9c8ea30e67045e9bb48382ebfb0c7252aae6ab43d3a WatchSource:0}: Error finding container 019d9353f1ff97e533ebc9c8ea30e67045e9bb48382ebfb0c7252aae6ab43d3a: Status 404 returned error can't find the container with id 019d9353f1ff97e533ebc9c8ea30e67045e9bb48382ebfb0c7252aae6ab43d3a
Sep 16 13:07:18.642520 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:18.642485    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f0722f-57b85b59f7-2j54q" event={"ID":"2389a693-9d04-4a93-97df-0649e5f7941e","Type":"ContainerStarted","Data":"d5215370ddb8abf3bca81b58d1dcb87e9eb002eb5c2d6960f18029e7081df6b9"}
Sep 16 13:07:18.642743 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:18.642724    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f0722f-57b85b59f7-2j54q"
Sep 16 13:07:18.642816 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:18.642765    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f0722f-57b85b59f7-2j54q" event={"ID":"2389a693-9d04-4a93-97df-0649e5f7941e","Type":"ContainerStarted","Data":"019d9353f1ff97e533ebc9c8ea30e67045e9bb48382ebfb0c7252aae6ab43d3a"}
Sep 16 13:07:18.644141 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:18.644115    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0722f-57df84c966-jqqj2" event={"ID":"4e1cd797-8cf2-46ec-8779-30f3fa51d5da","Type":"ContainerStarted","Data":"ff4973bab41608f7b7582aeb7b2d5ba683a8a9cfaea629cb2f123cc5d9ed03ad"}
Sep 16 13:07:18.644262 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:18.644164    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0722f-57df84c966-jqqj2" event={"ID":"4e1cd797-8cf2-46ec-8779-30f3fa51d5da","Type":"ContainerStarted","Data":"d94f12e2d6239c613111ef10329e6c63981efed34883efff23bf1d83a7a97c04"}
Sep 16 13:07:18.644262 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:18.644187    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0722f-57df84c966-jqqj2"
Sep 16 13:07:18.660298 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:18.660229    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f0722f-57b85b59f7-2j54q" podStartSLOduration=1.660216744 podStartE2EDuration="1.660216744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:07:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:07:18.659195808 +0000 UTC m=+1745.106428450" watchObservedRunningTime="2026-09-16 13:07:18.660216744 +0000 UTC m=+1745.107449388"
Sep 16 13:07:18.676413 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:18.676367    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0722f-57df84c966-jqqj2" podStartSLOduration=1.676353529 podStartE2EDuration="1.676353529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:07:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:07:18.675578791 +0000 UTC m=+1745.122811427" watchObservedRunningTime="2026-09-16 13:07:18.676353529 +0000 UTC m=+1745.123586171"
Sep 16 13:07:29.651965 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:29.651933    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0722f-57df84c966-jqqj2"
Sep 16 13:07:29.652528 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:29.652000    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f0722f-57b85b59f7-2j54q"
Sep 16 13:07:53.475377 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:53.475344    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-94f3e1-667bc66ccc-8cbjn"]
Sep 16 13:07:53.480321 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:53.480298    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-94f3e1-667bc66ccc-8cbjn"
Sep 16 13:07:53.483330 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:53.483308    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-94f3e1-dockercfg-cghbn\""
Sep 16 13:07:53.483470 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:53.483348    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-94f3e1\""
Sep 16 13:07:53.490740 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:53.490715    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-94f3e1-667bc66ccc-8cbjn"]
Sep 16 13:07:53.655948 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:53.655912    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwpvd\" (UniqueName: \"kubernetes.io/projected/9adac33b-8012-489a-a305-06b82f1da223-kube-api-access-hwpvd\") pod \"payload-processing-e2e-api-b-94f3e1-667bc66ccc-8cbjn\" (UID: \"9adac33b-8012-489a-a305-06b82f1da223\") " pod="openshift-ingress/payload-processing-e2e-api-b-94f3e1-667bc66ccc-8cbjn"
Sep 16 13:07:53.656147 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:53.655975    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9adac33b-8012-489a-a305-06b82f1da223-plugins-config-volume\") pod \"payload-processing-e2e-api-b-94f3e1-667bc66ccc-8cbjn\" (UID: \"9adac33b-8012-489a-a305-06b82f1da223\") " pod="openshift-ingress/payload-processing-e2e-api-b-94f3e1-667bc66ccc-8cbjn"
Sep 16 13:07:53.757453 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:53.757412    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwpvd\" (UniqueName: \"kubernetes.io/projected/9adac33b-8012-489a-a305-06b82f1da223-kube-api-access-hwpvd\") pod \"payload-processing-e2e-api-b-94f3e1-667bc66ccc-8cbjn\" (UID: \"9adac33b-8012-489a-a305-06b82f1da223\") " pod="openshift-ingress/payload-processing-e2e-api-b-94f3e1-667bc66ccc-8cbjn"
Sep 16 13:07:53.757639 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:53.757504    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9adac33b-8012-489a-a305-06b82f1da223-plugins-config-volume\") pod \"payload-processing-e2e-api-b-94f3e1-667bc66ccc-8cbjn\" (UID: \"9adac33b-8012-489a-a305-06b82f1da223\") " pod="openshift-ingress/payload-processing-e2e-api-b-94f3e1-667bc66ccc-8cbjn"
Sep 16 13:07:53.758221 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:53.758203    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9adac33b-8012-489a-a305-06b82f1da223-plugins-config-volume\") pod \"payload-processing-e2e-api-b-94f3e1-667bc66ccc-8cbjn\" (UID: \"9adac33b-8012-489a-a305-06b82f1da223\") " pod="openshift-ingress/payload-processing-e2e-api-b-94f3e1-667bc66ccc-8cbjn"
Sep 16 13:07:53.769161 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:53.769134    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwpvd\" (UniqueName: \"kubernetes.io/projected/9adac33b-8012-489a-a305-06b82f1da223-kube-api-access-hwpvd\") pod \"payload-processing-e2e-api-b-94f3e1-667bc66ccc-8cbjn\" (UID: \"9adac33b-8012-489a-a305-06b82f1da223\") " pod="openshift-ingress/payload-processing-e2e-api-b-94f3e1-667bc66ccc-8cbjn"
Sep 16 13:07:53.794071 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:53.794042    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-94f3e1-667bc66ccc-8cbjn"
Sep 16 13:07:54.073273 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:54.073246    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-94f3e1-667bc66ccc-8cbjn"]
Sep 16 13:07:54.075011 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:07:54.074969    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9adac33b_8012_489a_a305_06b82f1da223.slice/crio-37b6832de394776fd18701d8dfe9e68f581dd399d0b674e2f32b83a8f06826c3 WatchSource:0}: Error finding container 37b6832de394776fd18701d8dfe9e68f581dd399d0b674e2f32b83a8f06826c3: Status 404 returned error can't find the container with id 37b6832de394776fd18701d8dfe9e68f581dd399d0b674e2f32b83a8f06826c3
Sep 16 13:07:54.864643 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:54.864611    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-94f3e1-667bc66ccc-8cbjn" event={"ID":"9adac33b-8012-489a-a305-06b82f1da223","Type":"ContainerStarted","Data":"fbeec1bea8cf91d98898625e0d7cc3609f563f7da2b4d91885ddffbcab765ec0"}
Sep 16 13:07:54.865093 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:54.864662    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-94f3e1-667bc66ccc-8cbjn" event={"ID":"9adac33b-8012-489a-a305-06b82f1da223","Type":"ContainerStarted","Data":"37b6832de394776fd18701d8dfe9e68f581dd399d0b674e2f32b83a8f06826c3"}
Sep 16 13:07:54.865093 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:54.864690    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-94f3e1-667bc66ccc-8cbjn"
Sep 16 13:07:54.884389 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:07:54.884349    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-94f3e1-667bc66ccc-8cbjn" podStartSLOduration=1.884332545 podStartE2EDuration="1.884332545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:07:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:07:54.881253982 +0000 UTC m=+1781.328486637" watchObservedRunningTime="2026-09-16 13:07:54.884332545 +0000 UTC m=+1781.331565186"
Sep 16 13:08:05.872675 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:05.872595    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-94f3e1-667bc66ccc-8cbjn"
Sep 16 13:08:14.184868 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:14.184833    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vbt8d_0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174/console-operator/1.log"
Sep 16 13:08:14.191765 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:14.191740    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vbt8d_0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174/console-operator/1.log"
Sep 16 13:08:14.983732 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:14.983695    2577 scope.go:117] "RemoveContainer" containerID="2f5ef2541085b0c7adc73674cc5c01ac7aa601e6ff303fdd50109cc24095a608"
Sep 16 13:08:20.057180 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:20.057146    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0722f-djxvq"]
Sep 16 13:08:20.060932 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:20.060917    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0722f-djxvq"
Sep 16 13:08:20.063791 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:20.063753    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-64zph\""
Sep 16 13:08:20.063942 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:20.063884    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 13:08:20.066892 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:20.066866    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0722f-djxvq"]
Sep 16 13:08:20.125963 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:20.125931    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1e3fcf7-669f-464f-a956-782b490c6ed5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f0722f-djxvq\" (UID: \"a1e3fcf7-669f-464f-a956-782b490c6ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0722f-djxvq"
Sep 16 13:08:20.227177 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:20.227138    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1e3fcf7-669f-464f-a956-782b490c6ed5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f0722f-djxvq\" (UID: \"a1e3fcf7-669f-464f-a956-782b490c6ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0722f-djxvq"
Sep 16 13:08:20.227757 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:20.227738    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1e3fcf7-669f-464f-a956-782b490c6ed5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f0722f-djxvq\" (UID: \"a1e3fcf7-669f-464f-a956-782b490c6ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0722f-djxvq"
Sep 16 13:08:20.376078 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:20.375975    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0722f-djxvq"
Sep 16 13:08:20.652654 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:20.652624    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0722f-djxvq"]
Sep 16 13:08:20.654357 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:08:20.654330    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e3fcf7_669f_464f_a956_782b490c6ed5.slice/crio-a31c6f41b9e1572f146d983c3952b4ff962b3960f6ffee0ab3414dcb424139d8 WatchSource:0}: Error finding container a31c6f41b9e1572f146d983c3952b4ff962b3960f6ffee0ab3414dcb424139d8: Status 404 returned error can't find the container with id a31c6f41b9e1572f146d983c3952b4ff962b3960f6ffee0ab3414dcb424139d8
Sep 16 13:08:21.030343 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:21.030305    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0722f-djxvq" event={"ID":"a1e3fcf7-669f-464f-a956-782b490c6ed5","Type":"ContainerStarted","Data":"e285bd0d6fab27393493fe9068afc0fe1e83fe2327eed89b33f5be92366e74b7"}
Sep 16 13:08:21.030481 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:21.030369    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0722f-djxvq" event={"ID":"a1e3fcf7-669f-464f-a956-782b490c6ed5","Type":"ContainerStarted","Data":"a31c6f41b9e1572f146d983c3952b4ff962b3960f6ffee0ab3414dcb424139d8"}
Sep 16 13:08:21.048536 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:21.048492    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0722f-djxvq" podStartSLOduration=1.048478658 podStartE2EDuration="1.048478658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:08:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:08:21.04655092 +0000 UTC m=+1807.493783563" watchObservedRunningTime="2026-09-16 13:08:21.048478658 +0000 UTC m=+1807.495711345"
Sep 16 13:08:22.038359 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:22.038324    2577 generic.go:356] "Generic (PLEG): container finished" podID="a1e3fcf7-669f-464f-a956-782b490c6ed5" containerID="e285bd0d6fab27393493fe9068afc0fe1e83fe2327eed89b33f5be92366e74b7" exitCode=0
Sep 16 13:08:22.038744 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:22.038379    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0722f-djxvq" event={"ID":"a1e3fcf7-669f-464f-a956-782b490c6ed5","Type":"ContainerDied","Data":"e285bd0d6fab27393493fe9068afc0fe1e83fe2327eed89b33f5be92366e74b7"}
Sep 16 13:08:23.188762 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:23.188729    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0722f-djxvq"
Sep 16 13:08:23.257123 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:23.257087    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1e3fcf7-669f-464f-a956-782b490c6ed5-maas-api-service-ca\") pod \"a1e3fcf7-669f-464f-a956-782b490c6ed5\" (UID: \"a1e3fcf7-669f-464f-a956-782b490c6ed5\") "
Sep 16 13:08:23.257526 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:23.257503    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1e3fcf7-669f-464f-a956-782b490c6ed5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a1e3fcf7-669f-464f-a956-782b490c6ed5" (UID: "a1e3fcf7-669f-464f-a956-782b490c6ed5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:08:23.359367 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:23.359280    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1e3fcf7-669f-464f-a956-782b490c6ed5-maas-api-service-ca\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:08:24.052897 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:24.052869    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0722f-djxvq"
Sep 16 13:08:24.055318 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:24.055288    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0722f-djxvq" event={"ID":"a1e3fcf7-669f-464f-a956-782b490c6ed5","Type":"ContainerDied","Data":"a31c6f41b9e1572f146d983c3952b4ff962b3960f6ffee0ab3414dcb424139d8"}
Sep 16 13:08:24.055451 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:24.055337    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a31c6f41b9e1572f146d983c3952b4ff962b3960f6ffee0ab3414dcb424139d8"
Sep 16 13:08:36.026747 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:36.026709    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f0722f-57b85b59f7-2j54q"]
Sep 16 13:08:36.027267 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:36.026957    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f0722f-57b85b59f7-2j54q" podUID="2389a693-9d04-4a93-97df-0649e5f7941e" containerName="payload-processing" containerID="cri-o://d5215370ddb8abf3bca81b58d1dcb87e9eb002eb5c2d6960f18029e7081df6b9" gracePeriod=30
Sep 16 13:08:36.036178 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:36.036148    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f0722f-57df84c966-jqqj2"]
Sep 16 13:08:36.036408 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:36.036378    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0722f-57df84c966-jqqj2" podUID="4e1cd797-8cf2-46ec-8779-30f3fa51d5da" containerName="payload-pre-processing" containerID="cri-o://ff4973bab41608f7b7582aeb7b2d5ba683a8a9cfaea629cb2f123cc5d9ed03ad" gracePeriod=30
Sep 16 13:08:36.223952 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:36.223888    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0722f-djxvq"]
Sep 16 13:08:36.226400 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:36.226372    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0722f-djxvq"]
Sep 16 13:08:36.296405 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:36.296376    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f0722f-57b85b59f7-2j54q"
Sep 16 13:08:36.329197 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:36.329175    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0722f-57df84c966-jqqj2"
Sep 16 13:08:36.390368 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:36.390340    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e1cd797-8cf2-46ec-8779-30f3fa51d5da-plugins-config-volume\") pod \"4e1cd797-8cf2-46ec-8779-30f3fa51d5da\" (UID: \"4e1cd797-8cf2-46ec-8779-30f3fa51d5da\") "
Sep 16 13:08:36.390565 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:36.390381    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znp66\" (UniqueName: \"kubernetes.io/projected/4e1cd797-8cf2-46ec-8779-30f3fa51d5da-kube-api-access-znp66\") pod \"4e1cd797-8cf2-46ec-8779-30f3fa51d5da\" (UID: \"4e1cd797-8cf2-46ec-8779-30f3fa51d5da\") "
Sep 16 13:08:36.390565 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:36.390418    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2389a693-9d04-4a93-97df-0649e5f7941e-plugins-config-volume\") pod \"2389a693-9d04-4a93-97df-0649e5f7941e\" (UID: \"2389a693-9d04-4a93-97df-0649e5f7941e\") "
Sep 16 13:08:36.390565 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:36.390477    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h87tz\" (UniqueName: \"kubernetes.io/projected/2389a693-9d04-4a93-97df-0649e5f7941e-kube-api-access-h87tz\") pod \"2389a693-9d04-4a93-97df-0649e5f7941e\" (UID: \"2389a693-9d04-4a93-97df-0649e5f7941e\") "
Sep 16 13:08:36.390863 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:36.390832    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e1cd797-8cf2-46ec-8779-30f3fa51d5da-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4e1cd797-8cf2-46ec-8779-30f3fa51d5da" (UID: "4e1cd797-8cf2-46ec-8779-30f3fa51d5da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:08:36.391320 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:36.391293    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2389a693-9d04-4a93-97df-0649e5f7941e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2389a693-9d04-4a93-97df-0649e5f7941e" (UID: "2389a693-9d04-4a93-97df-0649e5f7941e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:08:36.393717 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:36.393694    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2389a693-9d04-4a93-97df-0649e5f7941e-kube-api-access-h87tz" (OuterVolumeSpecName: "kube-api-access-h87tz") pod "2389a693-9d04-4a93-97df-0649e5f7941e" (UID: "2389a693-9d04-4a93-97df-0649e5f7941e"). InnerVolumeSpecName "kube-api-access-h87tz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:08:36.393812 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:36.393724    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e1cd797-8cf2-46ec-8779-30f3fa51d5da-kube-api-access-znp66" (OuterVolumeSpecName: "kube-api-access-znp66") pod "4e1cd797-8cf2-46ec-8779-30f3fa51d5da" (UID: "4e1cd797-8cf2-46ec-8779-30f3fa51d5da"). InnerVolumeSpecName "kube-api-access-znp66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:08:36.491646 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:36.491609    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e1cd797-8cf2-46ec-8779-30f3fa51d5da-plugins-config-volume\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:08:36.491646 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:36.491647    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znp66\" (UniqueName: \"kubernetes.io/projected/4e1cd797-8cf2-46ec-8779-30f3fa51d5da-kube-api-access-znp66\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:08:36.491828 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:36.491662    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2389a693-9d04-4a93-97df-0649e5f7941e-plugins-config-volume\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:08:36.491828 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:36.491675    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h87tz\" (UniqueName: \"kubernetes.io/projected/2389a693-9d04-4a93-97df-0649e5f7941e-kube-api-access-h87tz\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:08:37.136571 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:37.136535    2577 generic.go:356] "Generic (PLEG): container finished" podID="2389a693-9d04-4a93-97df-0649e5f7941e" containerID="d5215370ddb8abf3bca81b58d1dcb87e9eb002eb5c2d6960f18029e7081df6b9" exitCode=0
Sep 16 13:08:37.137128 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:37.136594    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f0722f-57b85b59f7-2j54q"
Sep 16 13:08:37.137128 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:37.136625    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f0722f-57b85b59f7-2j54q" event={"ID":"2389a693-9d04-4a93-97df-0649e5f7941e","Type":"ContainerDied","Data":"d5215370ddb8abf3bca81b58d1dcb87e9eb002eb5c2d6960f18029e7081df6b9"}
Sep 16 13:08:37.137128 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:37.136694    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f0722f-57b85b59f7-2j54q" event={"ID":"2389a693-9d04-4a93-97df-0649e5f7941e","Type":"ContainerDied","Data":"019d9353f1ff97e533ebc9c8ea30e67045e9bb48382ebfb0c7252aae6ab43d3a"}
Sep 16 13:08:37.137128 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:37.136729    2577 scope.go:117] "RemoveContainer" containerID="d5215370ddb8abf3bca81b58d1dcb87e9eb002eb5c2d6960f18029e7081df6b9"
Sep 16 13:08:37.138246 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:37.138096    2577 generic.go:356] "Generic (PLEG): container finished" podID="4e1cd797-8cf2-46ec-8779-30f3fa51d5da" containerID="ff4973bab41608f7b7582aeb7b2d5ba683a8a9cfaea629cb2f123cc5d9ed03ad" exitCode=0
Sep 16 13:08:37.138246 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:37.138167    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0722f-57df84c966-jqqj2"
Sep 16 13:08:37.138246 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:37.138179    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0722f-57df84c966-jqqj2" event={"ID":"4e1cd797-8cf2-46ec-8779-30f3fa51d5da","Type":"ContainerDied","Data":"ff4973bab41608f7b7582aeb7b2d5ba683a8a9cfaea629cb2f123cc5d9ed03ad"}
Sep 16 13:08:37.138246 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:37.138213    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0722f-57df84c966-jqqj2" event={"ID":"4e1cd797-8cf2-46ec-8779-30f3fa51d5da","Type":"ContainerDied","Data":"d94f12e2d6239c613111ef10329e6c63981efed34883efff23bf1d83a7a97c04"}
Sep 16 13:08:37.147977 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:37.147960    2577 scope.go:117] "RemoveContainer" containerID="d5215370ddb8abf3bca81b58d1dcb87e9eb002eb5c2d6960f18029e7081df6b9"
Sep 16 13:08:37.148381 ip-10-0-133-190 kubenswrapper[2577]: E0916 13:08:37.148320    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5215370ddb8abf3bca81b58d1dcb87e9eb002eb5c2d6960f18029e7081df6b9\": container with ID starting with d5215370ddb8abf3bca81b58d1dcb87e9eb002eb5c2d6960f18029e7081df6b9 not found: ID does not exist" containerID="d5215370ddb8abf3bca81b58d1dcb87e9eb002eb5c2d6960f18029e7081df6b9"
Sep 16 13:08:37.148381 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:37.148352    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5215370ddb8abf3bca81b58d1dcb87e9eb002eb5c2d6960f18029e7081df6b9"} err="failed to get container status \"d5215370ddb8abf3bca81b58d1dcb87e9eb002eb5c2d6960f18029e7081df6b9\": rpc error: code = NotFound desc = could not find container \"d5215370ddb8abf3bca81b58d1dcb87e9eb002eb5c2d6960f18029e7081df6b9\": container with ID starting with d5215370ddb8abf3bca81b58d1dcb87e9eb002eb5c2d6960f18029e7081df6b9 not found: ID does not exist"
Sep 16 13:08:37.148381 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:37.148376    2577 scope.go:117] "RemoveContainer" containerID="ff4973bab41608f7b7582aeb7b2d5ba683a8a9cfaea629cb2f123cc5d9ed03ad"
Sep 16 13:08:37.156936 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:37.156919    2577 scope.go:117] "RemoveContainer" containerID="ff4973bab41608f7b7582aeb7b2d5ba683a8a9cfaea629cb2f123cc5d9ed03ad"
Sep 16 13:08:37.157223 ip-10-0-133-190 kubenswrapper[2577]: E0916 13:08:37.157205    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff4973bab41608f7b7582aeb7b2d5ba683a8a9cfaea629cb2f123cc5d9ed03ad\": container with ID starting with ff4973bab41608f7b7582aeb7b2d5ba683a8a9cfaea629cb2f123cc5d9ed03ad not found: ID does not exist" containerID="ff4973bab41608f7b7582aeb7b2d5ba683a8a9cfaea629cb2f123cc5d9ed03ad"
Sep 16 13:08:37.157278 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:37.157231    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff4973bab41608f7b7582aeb7b2d5ba683a8a9cfaea629cb2f123cc5d9ed03ad"} err="failed to get container status \"ff4973bab41608f7b7582aeb7b2d5ba683a8a9cfaea629cb2f123cc5d9ed03ad\": rpc error: code = NotFound desc = could not find container \"ff4973bab41608f7b7582aeb7b2d5ba683a8a9cfaea629cb2f123cc5d9ed03ad\": container with ID starting with ff4973bab41608f7b7582aeb7b2d5ba683a8a9cfaea629cb2f123cc5d9ed03ad not found: ID does not exist"
Sep 16 13:08:37.169329 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:37.169308    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f0722f-57b85b59f7-2j54q"]
Sep 16 13:08:37.172527 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:37.172506    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f0722f-57b85b59f7-2j54q"]
Sep 16 13:08:37.185014 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:37.184989    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f0722f-57df84c966-jqqj2"]
Sep 16 13:08:37.192507 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:37.192476    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f0722f-57df84c966-jqqj2"]
Sep 16 13:08:38.055428 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:38.055394    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2389a693-9d04-4a93-97df-0649e5f7941e" path="/var/lib/kubelet/pods/2389a693-9d04-4a93-97df-0649e5f7941e/volumes"
Sep 16 13:08:38.055874 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:38.055854    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e1cd797-8cf2-46ec-8779-30f3fa51d5da" path="/var/lib/kubelet/pods/4e1cd797-8cf2-46ec-8779-30f3fa51d5da/volumes"
Sep 16 13:08:38.056265 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:38.056247    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1e3fcf7-669f-464f-a956-782b490c6ed5" path="/var/lib/kubelet/pods/a1e3fcf7-669f-464f-a956-782b490c6ed5/volumes"
Sep 16 13:08:43.549157 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:43.549123    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-94f3e1-rgj6v"]
Sep 16 13:08:43.549621 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:43.549602    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e1cd797-8cf2-46ec-8779-30f3fa51d5da" containerName="payload-pre-processing"
Sep 16 13:08:43.549621 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:43.549622    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e1cd797-8cf2-46ec-8779-30f3fa51d5da" containerName="payload-pre-processing"
Sep 16 13:08:43.549708 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:43.549651    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2389a693-9d04-4a93-97df-0649e5f7941e" containerName="payload-processing"
Sep 16 13:08:43.549708 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:43.549657    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2389a693-9d04-4a93-97df-0649e5f7941e" containerName="payload-processing"
Sep 16 13:08:43.549708 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:43.549664    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1e3fcf7-669f-464f-a956-782b490c6ed5" containerName="revoke-keys"
Sep 16 13:08:43.549708 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:43.549669    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1e3fcf7-669f-464f-a956-782b490c6ed5" containerName="revoke-keys"
Sep 16 13:08:43.549846 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:43.549738    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e1cd797-8cf2-46ec-8779-30f3fa51d5da" containerName="payload-pre-processing"
Sep 16 13:08:43.549846 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:43.549744    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1e3fcf7-669f-464f-a956-782b490c6ed5" containerName="revoke-keys"
Sep 16 13:08:43.549846 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:43.549754    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2389a693-9d04-4a93-97df-0649e5f7941e" containerName="payload-processing"
Sep 16 13:08:43.554339 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:43.554322    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-94f3e1-rgj6v"
Sep 16 13:08:43.556827 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:43.556802    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-64zph\""
Sep 16 13:08:43.556972 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:43.556846    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 13:08:43.561245 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:43.561224    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-94f3e1-rgj6v"]
Sep 16 13:08:43.674947 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:43.674910    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/709f7586-cd5f-4d17-90c7-6a97e8a45852-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-94f3e1-rgj6v\" (UID: \"709f7586-cd5f-4d17-90c7-6a97e8a45852\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-94f3e1-rgj6v"
Sep 16 13:08:43.775736 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:43.775702    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/709f7586-cd5f-4d17-90c7-6a97e8a45852-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-94f3e1-rgj6v\" (UID: \"709f7586-cd5f-4d17-90c7-6a97e8a45852\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-94f3e1-rgj6v"
Sep 16 13:08:43.776395 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:43.776376    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/709f7586-cd5f-4d17-90c7-6a97e8a45852-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-94f3e1-rgj6v\" (UID: \"709f7586-cd5f-4d17-90c7-6a97e8a45852\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-94f3e1-rgj6v"
Sep 16 13:08:43.867814 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:43.867729    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-94f3e1-rgj6v"
Sep 16 13:08:44.146671 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:44.146646    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-94f3e1-rgj6v"]
Sep 16 13:08:44.148455 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:08:44.148429    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod709f7586_cd5f_4d17_90c7_6a97e8a45852.slice/crio-2a0942fe293fe3969c573988a2b7585a1f885795fc9fce2833c890a60807e832 WatchSource:0}: Error finding container 2a0942fe293fe3969c573988a2b7585a1f885795fc9fce2833c890a60807e832: Status 404 returned error can't find the container with id 2a0942fe293fe3969c573988a2b7585a1f885795fc9fce2833c890a60807e832
Sep 16 13:08:44.186652 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:44.186622    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-94f3e1-rgj6v" event={"ID":"709f7586-cd5f-4d17-90c7-6a97e8a45852","Type":"ContainerStarted","Data":"2a0942fe293fe3969c573988a2b7585a1f885795fc9fce2833c890a60807e832"}
Sep 16 13:08:45.195009 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:45.194907    2577 generic.go:356] "Generic (PLEG): container finished" podID="709f7586-cd5f-4d17-90c7-6a97e8a45852" containerID="73543931499ac12c8dfd9bbe5a31c29b050296f4c62c1f5f99ca9a4951b91bf5" exitCode=0
Sep 16 13:08:45.195507 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:45.194998    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-94f3e1-rgj6v" event={"ID":"709f7586-cd5f-4d17-90c7-6a97e8a45852","Type":"ContainerDied","Data":"73543931499ac12c8dfd9bbe5a31c29b050296f4c62c1f5f99ca9a4951b91bf5"}
Sep 16 13:08:46.351293 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:46.351273    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-94f3e1-rgj6v"
Sep 16 13:08:46.405964 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:46.405932    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/709f7586-cd5f-4d17-90c7-6a97e8a45852-maas-api-service-ca\") pod \"709f7586-cd5f-4d17-90c7-6a97e8a45852\" (UID: \"709f7586-cd5f-4d17-90c7-6a97e8a45852\") "
Sep 16 13:08:46.406316 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:46.406293    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/709f7586-cd5f-4d17-90c7-6a97e8a45852-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "709f7586-cd5f-4d17-90c7-6a97e8a45852" (UID: "709f7586-cd5f-4d17-90c7-6a97e8a45852"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:08:46.406444 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:46.406397    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/709f7586-cd5f-4d17-90c7-6a97e8a45852-maas-api-service-ca\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:08:47.208191 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:47.208153    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-94f3e1-rgj6v"
Sep 16 13:08:47.208372 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:47.208208    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-94f3e1-rgj6v" event={"ID":"709f7586-cd5f-4d17-90c7-6a97e8a45852","Type":"ContainerDied","Data":"2a0942fe293fe3969c573988a2b7585a1f885795fc9fce2833c890a60807e832"}
Sep 16 13:08:47.208372 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:47.208255    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a0942fe293fe3969c573988a2b7585a1f885795fc9fce2833c890a60807e832"
Sep 16 13:08:59.178413 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:59.178371    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-94f3e1-667bc66ccc-8cbjn"]
Sep 16 13:08:59.179458 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:59.179359    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-94f3e1-667bc66ccc-8cbjn" podUID="9adac33b-8012-489a-a305-06b82f1da223" containerName="payload-processing" containerID="cri-o://fbeec1bea8cf91d98898625e0d7cc3609f563f7da2b4d91885ddffbcab765ec0" gracePeriod=30
Sep 16 13:08:59.340898 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:59.340840    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-94f3e1-rgj6v"]
Sep 16 13:08:59.343136 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:59.343105    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-94f3e1-rgj6v"]
Sep 16 13:08:59.445011 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:59.444954    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-94f3e1-667bc66ccc-8cbjn"
Sep 16 13:08:59.542886 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:59.542856    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwpvd\" (UniqueName: \"kubernetes.io/projected/9adac33b-8012-489a-a305-06b82f1da223-kube-api-access-hwpvd\") pod \"9adac33b-8012-489a-a305-06b82f1da223\" (UID: \"9adac33b-8012-489a-a305-06b82f1da223\") "
Sep 16 13:08:59.543109 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:59.542906    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9adac33b-8012-489a-a305-06b82f1da223-plugins-config-volume\") pod \"9adac33b-8012-489a-a305-06b82f1da223\" (UID: \"9adac33b-8012-489a-a305-06b82f1da223\") "
Sep 16 13:08:59.543332 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:59.543305    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9adac33b-8012-489a-a305-06b82f1da223-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9adac33b-8012-489a-a305-06b82f1da223" (UID: "9adac33b-8012-489a-a305-06b82f1da223"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:08:59.545192 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:59.545174    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9adac33b-8012-489a-a305-06b82f1da223-kube-api-access-hwpvd" (OuterVolumeSpecName: "kube-api-access-hwpvd") pod "9adac33b-8012-489a-a305-06b82f1da223" (UID: "9adac33b-8012-489a-a305-06b82f1da223"). InnerVolumeSpecName "kube-api-access-hwpvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:08:59.644356 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:59.644321    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwpvd\" (UniqueName: \"kubernetes.io/projected/9adac33b-8012-489a-a305-06b82f1da223-kube-api-access-hwpvd\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:08:59.644356 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:08:59.644349    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9adac33b-8012-489a-a305-06b82f1da223-plugins-config-volume\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:00.056134 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:00.056096    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="709f7586-cd5f-4d17-90c7-6a97e8a45852" path="/var/lib/kubelet/pods/709f7586-cd5f-4d17-90c7-6a97e8a45852/volumes"
Sep 16 13:09:00.288197 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:00.287962    2577 generic.go:356] "Generic (PLEG): container finished" podID="9adac33b-8012-489a-a305-06b82f1da223" containerID="fbeec1bea8cf91d98898625e0d7cc3609f563f7da2b4d91885ddffbcab765ec0" exitCode=0
Sep 16 13:09:00.288197 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:00.288055    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-94f3e1-667bc66ccc-8cbjn" event={"ID":"9adac33b-8012-489a-a305-06b82f1da223","Type":"ContainerDied","Data":"fbeec1bea8cf91d98898625e0d7cc3609f563f7da2b4d91885ddffbcab765ec0"}
Sep 16 13:09:00.288197 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:00.288125    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-94f3e1-667bc66ccc-8cbjn" event={"ID":"9adac33b-8012-489a-a305-06b82f1da223","Type":"ContainerDied","Data":"37b6832de394776fd18701d8dfe9e68f581dd399d0b674e2f32b83a8f06826c3"}
Sep 16 13:09:00.288197 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:00.288159    2577 scope.go:117] "RemoveContainer" containerID="fbeec1bea8cf91d98898625e0d7cc3609f563f7da2b4d91885ddffbcab765ec0"
Sep 16 13:09:00.289782 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:00.289760    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-94f3e1-667bc66ccc-8cbjn"
Sep 16 13:09:00.302137 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:00.302116    2577 scope.go:117] "RemoveContainer" containerID="fbeec1bea8cf91d98898625e0d7cc3609f563f7da2b4d91885ddffbcab765ec0"
Sep 16 13:09:00.302508 ip-10-0-133-190 kubenswrapper[2577]: E0916 13:09:00.302469    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbeec1bea8cf91d98898625e0d7cc3609f563f7da2b4d91885ddffbcab765ec0\": container with ID starting with fbeec1bea8cf91d98898625e0d7cc3609f563f7da2b4d91885ddffbcab765ec0 not found: ID does not exist" containerID="fbeec1bea8cf91d98898625e0d7cc3609f563f7da2b4d91885ddffbcab765ec0"
Sep 16 13:09:00.302623 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:00.302515    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbeec1bea8cf91d98898625e0d7cc3609f563f7da2b4d91885ddffbcab765ec0"} err="failed to get container status \"fbeec1bea8cf91d98898625e0d7cc3609f563f7da2b4d91885ddffbcab765ec0\": rpc error: code = NotFound desc = could not find container \"fbeec1bea8cf91d98898625e0d7cc3609f563f7da2b4d91885ddffbcab765ec0\": container with ID starting with fbeec1bea8cf91d98898625e0d7cc3609f563f7da2b4d91885ddffbcab765ec0 not found: ID does not exist"
Sep 16 13:09:00.308122 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:00.308066    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-94f3e1-667bc66ccc-8cbjn"]
Sep 16 13:09:00.314271 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:00.314250    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-94f3e1-667bc66ccc-8cbjn"]
Sep 16 13:09:02.055949 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:02.055914    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9adac33b-8012-489a-a305-06b82f1da223" path="/var/lib/kubelet/pods/9adac33b-8012-489a-a305-06b82f1da223/volumes"
Sep 16 13:09:06.566800 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:06.566765    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-863d9d-dfq27"]
Sep 16 13:09:06.567249 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:06.567235    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="709f7586-cd5f-4d17-90c7-6a97e8a45852" containerName="revoke-keys"
Sep 16 13:09:06.567297 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:06.567252    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="709f7586-cd5f-4d17-90c7-6a97e8a45852" containerName="revoke-keys"
Sep 16 13:09:06.567297 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:06.567266    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9adac33b-8012-489a-a305-06b82f1da223" containerName="payload-processing"
Sep 16 13:09:06.567297 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:06.567273    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9adac33b-8012-489a-a305-06b82f1da223" containerName="payload-processing"
Sep 16 13:09:06.567392 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:06.567368    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9adac33b-8012-489a-a305-06b82f1da223" containerName="payload-processing"
Sep 16 13:09:06.567392 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:06.567378    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="709f7586-cd5f-4d17-90c7-6a97e8a45852" containerName="revoke-keys"
Sep 16 13:09:06.572040 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:06.572009    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-863d9d-dfq27"
Sep 16 13:09:06.574910 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:06.574890    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 13:09:06.575195 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:06.575167    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-64zph\""
Sep 16 13:09:06.579453 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:06.579423    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-863d9d-dfq27"]
Sep 16 13:09:06.615847 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:06.615822    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91a78016-5b3a-4b46-98bf-cc7c58c3213e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-863d9d-dfq27\" (UID: \"91a78016-5b3a-4b46-98bf-cc7c58c3213e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-863d9d-dfq27"
Sep 16 13:09:06.716730 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:06.716694    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91a78016-5b3a-4b46-98bf-cc7c58c3213e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-863d9d-dfq27\" (UID: \"91a78016-5b3a-4b46-98bf-cc7c58c3213e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-863d9d-dfq27"
Sep 16 13:09:06.717363 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:06.717341    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91a78016-5b3a-4b46-98bf-cc7c58c3213e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-863d9d-dfq27\" (UID: \"91a78016-5b3a-4b46-98bf-cc7c58c3213e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-863d9d-dfq27"
Sep 16 13:09:06.884912 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:06.884828    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-863d9d-dfq27"
Sep 16 13:09:07.161344 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:07.161322    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-863d9d-dfq27"]
Sep 16 13:09:07.162923 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:09:07.162891    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91a78016_5b3a_4b46_98bf_cc7c58c3213e.slice/crio-b9dcdb3931a860869629f44b44d81b79bf08225cc65e6ada4205f57e436a16aa WatchSource:0}: Error finding container b9dcdb3931a860869629f44b44d81b79bf08225cc65e6ada4205f57e436a16aa: Status 404 returned error can't find the container with id b9dcdb3931a860869629f44b44d81b79bf08225cc65e6ada4205f57e436a16aa
Sep 16 13:09:07.332476 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:07.332441    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-863d9d-dfq27" event={"ID":"91a78016-5b3a-4b46-98bf-cc7c58c3213e","Type":"ContainerStarted","Data":"b9dcdb3931a860869629f44b44d81b79bf08225cc65e6ada4205f57e436a16aa"}
Sep 16 13:09:08.339210 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:08.339176    2577 generic.go:356] "Generic (PLEG): container finished" podID="91a78016-5b3a-4b46-98bf-cc7c58c3213e" containerID="6c2eeca03991e3c1ca642ae8cc0d4a37fd8607dd52cd631407a89d3db2a799b4" exitCode=0
Sep 16 13:09:08.339624 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:08.339263    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-863d9d-dfq27" event={"ID":"91a78016-5b3a-4b46-98bf-cc7c58c3213e","Type":"ContainerDied","Data":"6c2eeca03991e3c1ca642ae8cc0d4a37fd8607dd52cd631407a89d3db2a799b4"}
Sep 16 13:09:09.491454 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:09.491425    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-863d9d-dfq27"
Sep 16 13:09:09.546651 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:09.546616    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91a78016-5b3a-4b46-98bf-cc7c58c3213e-maas-api-service-ca\") pod \"91a78016-5b3a-4b46-98bf-cc7c58c3213e\" (UID: \"91a78016-5b3a-4b46-98bf-cc7c58c3213e\") "
Sep 16 13:09:09.546975 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:09.546953    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91a78016-5b3a-4b46-98bf-cc7c58c3213e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "91a78016-5b3a-4b46-98bf-cc7c58c3213e" (UID: "91a78016-5b3a-4b46-98bf-cc7c58c3213e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:09:09.648007 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:09.647910    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91a78016-5b3a-4b46-98bf-cc7c58c3213e-maas-api-service-ca\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:10.352528 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:10.352432    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-863d9d-dfq27"
Sep 16 13:09:10.352528 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:10.352476    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-863d9d-dfq27" event={"ID":"91a78016-5b3a-4b46-98bf-cc7c58c3213e","Type":"ContainerDied","Data":"b9dcdb3931a860869629f44b44d81b79bf08225cc65e6ada4205f57e436a16aa"}
Sep 16 13:09:10.352718 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:10.352538    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9dcdb3931a860869629f44b44d81b79bf08225cc65e6ada4205f57e436a16aa"
Sep 16 13:09:17.031815 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:17.031777    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7d0256-kpksp"]
Sep 16 13:09:17.032267 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:17.032252    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91a78016-5b3a-4b46-98bf-cc7c58c3213e" containerName="revoke-keys"
Sep 16 13:09:17.032315 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:17.032269    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="91a78016-5b3a-4b46-98bf-cc7c58c3213e" containerName="revoke-keys"
Sep 16 13:09:17.032355 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:17.032351    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="91a78016-5b3a-4b46-98bf-cc7c58c3213e" containerName="revoke-keys"
Sep 16 13:09:17.035413 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:17.035397    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7d0256-kpksp"
Sep 16 13:09:17.038181 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:17.038159    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 13:09:17.038308 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:17.038196    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-64zph\""
Sep 16 13:09:17.043800 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:17.043778    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7d0256-kpksp"]
Sep 16 13:09:17.125827 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:17.125792    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ba05653-57c7-4e1d-8c93-c150188ff517-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-7d0256-kpksp\" (UID: \"6ba05653-57c7-4e1d-8c93-c150188ff517\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7d0256-kpksp"
Sep 16 13:09:17.226577 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:17.226536    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ba05653-57c7-4e1d-8c93-c150188ff517-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-7d0256-kpksp\" (UID: \"6ba05653-57c7-4e1d-8c93-c150188ff517\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7d0256-kpksp"
Sep 16 13:09:17.227224 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:17.227198    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ba05653-57c7-4e1d-8c93-c150188ff517-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-7d0256-kpksp\" (UID: \"6ba05653-57c7-4e1d-8c93-c150188ff517\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7d0256-kpksp"
Sep 16 13:09:17.347338 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:17.347253    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7d0256-kpksp"
Sep 16 13:09:17.623655 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:17.623632    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7d0256-kpksp"]
Sep 16 13:09:17.625668 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:09:17.625624    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ba05653_57c7_4e1d_8c93_c150188ff517.slice/crio-85443969c389c6f377fda7c92ecb5d0877811073b9648808d64b8c052122e3f8 WatchSource:0}: Error finding container 85443969c389c6f377fda7c92ecb5d0877811073b9648808d64b8c052122e3f8: Status 404 returned error can't find the container with id 85443969c389c6f377fda7c92ecb5d0877811073b9648808d64b8c052122e3f8
Sep 16 13:09:18.406298 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:18.406261    2577 generic.go:356] "Generic (PLEG): container finished" podID="6ba05653-57c7-4e1d-8c93-c150188ff517" containerID="3a1cec02ecf5eefc0dd5ab196e1aef23033ac3adce2125a65bc694f77e48e4ec" exitCode=0
Sep 16 13:09:18.406701 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:18.406359    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7d0256-kpksp" event={"ID":"6ba05653-57c7-4e1d-8c93-c150188ff517","Type":"ContainerDied","Data":"3a1cec02ecf5eefc0dd5ab196e1aef23033ac3adce2125a65bc694f77e48e4ec"}
Sep 16 13:09:18.406701 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:18.406413    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7d0256-kpksp" event={"ID":"6ba05653-57c7-4e1d-8c93-c150188ff517","Type":"ContainerStarted","Data":"85443969c389c6f377fda7c92ecb5d0877811073b9648808d64b8c052122e3f8"}
Sep 16 13:09:19.544773 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:19.544752    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7d0256-kpksp"
Sep 16 13:09:19.653520 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:19.653487    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ba05653-57c7-4e1d-8c93-c150188ff517-maas-api-service-ca\") pod \"6ba05653-57c7-4e1d-8c93-c150188ff517\" (UID: \"6ba05653-57c7-4e1d-8c93-c150188ff517\") "
Sep 16 13:09:19.653868 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:19.653849    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ba05653-57c7-4e1d-8c93-c150188ff517-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6ba05653-57c7-4e1d-8c93-c150188ff517" (UID: "6ba05653-57c7-4e1d-8c93-c150188ff517"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:09:19.754827 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:19.754800    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ba05653-57c7-4e1d-8c93-c150188ff517-maas-api-service-ca\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:20.418784 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:20.418702    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7d0256-kpksp"
Sep 16 13:09:20.418784 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:20.418724    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7d0256-kpksp" event={"ID":"6ba05653-57c7-4e1d-8c93-c150188ff517","Type":"ContainerDied","Data":"85443969c389c6f377fda7c92ecb5d0877811073b9648808d64b8c052122e3f8"}
Sep 16 13:09:20.418784 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:20.418769    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85443969c389c6f377fda7c92ecb5d0877811073b9648808d64b8c052122e3f8"
Sep 16 13:09:21.931276 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:21.931240    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-863d9d-dfq27"]
Sep 16 13:09:21.940660 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:21.940628    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-863d9d-dfq27"]
Sep 16 13:09:21.974968 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:21.974926    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7d0256/distinct-w4-7d0256-kserve-556b55b77-xcvvp"]
Sep 16 13:09:21.987089 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:21.987050    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7d0256/premium-w4-7d0256-kserve-69b8fcc96b-bcwl4"]
Sep 16 13:09:22.056282 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:22.056245    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91a78016-5b3a-4b46-98bf-cc7c58c3213e" path="/var/lib/kubelet/pods/91a78016-5b3a-4b46-98bf-cc7c58c3213e/volumes"
Sep 16 13:09:29.172397 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:29.172354    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e5926f-hfcpn"]
Sep 16 13:09:29.173011 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:29.172989    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ba05653-57c7-4e1d-8c93-c150188ff517" containerName="revoke-keys"
Sep 16 13:09:29.173145 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:29.173014    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ba05653-57c7-4e1d-8c93-c150188ff517" containerName="revoke-keys"
Sep 16 13:09:29.173145 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:29.173137    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ba05653-57c7-4e1d-8c93-c150188ff517" containerName="revoke-keys"
Sep 16 13:09:29.176633 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:29.176610    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e5926f-hfcpn"
Sep 16 13:09:29.179394 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:29.179378    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 13:09:29.179495 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:29.179480    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-64zph\""
Sep 16 13:09:29.184150 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:29.184127    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e5926f-hfcpn"]
Sep 16 13:09:29.359680 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:29.359642    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4559413e-b78b-4499-b85a-81ad046f306f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-e5926f-hfcpn\" (UID: \"4559413e-b78b-4499-b85a-81ad046f306f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e5926f-hfcpn"
Sep 16 13:09:29.461371 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:29.461276    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4559413e-b78b-4499-b85a-81ad046f306f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-e5926f-hfcpn\" (UID: \"4559413e-b78b-4499-b85a-81ad046f306f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e5926f-hfcpn"
Sep 16 13:09:29.461951 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:29.461922    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4559413e-b78b-4499-b85a-81ad046f306f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-e5926f-hfcpn\" (UID: \"4559413e-b78b-4499-b85a-81ad046f306f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e5926f-hfcpn"
Sep 16 13:09:29.489107 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:29.489077    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e5926f-hfcpn"
Sep 16 13:09:29.770849 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:29.770822    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e5926f-hfcpn"]
Sep 16 13:09:29.772707 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:09:29.772678    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4559413e_b78b_4499_b85a_81ad046f306f.slice/crio-24e01e31986318b5a5b4e3d5803e9b0c4e8e1949306c05bfbce99f84ec4883aa WatchSource:0}: Error finding container 24e01e31986318b5a5b4e3d5803e9b0c4e8e1949306c05bfbce99f84ec4883aa: Status 404 returned error can't find the container with id 24e01e31986318b5a5b4e3d5803e9b0c4e8e1949306c05bfbce99f84ec4883aa
Sep 16 13:09:30.478079 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:30.477968    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e5926f-hfcpn" event={"ID":"4559413e-b78b-4499-b85a-81ad046f306f","Type":"ContainerStarted","Data":"1364073e0b10fa0908c6bc1c1b4b3390d567aa54f826e1df2029e4ee4ac6bfa3"}
Sep 16 13:09:30.478079 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:30.478044    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e5926f-hfcpn" event={"ID":"4559413e-b78b-4499-b85a-81ad046f306f","Type":"ContainerStarted","Data":"24e01e31986318b5a5b4e3d5803e9b0c4e8e1949306c05bfbce99f84ec4883aa"}
Sep 16 13:09:30.495580 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:30.495529    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e5926f-hfcpn" podStartSLOduration=1.495513432 podStartE2EDuration="1.495513432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:09:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:30.4949642 +0000 UTC m=+1876.942196843" watchObservedRunningTime="2026-09-16 13:09:30.495513432 +0000 UTC m=+1876.942746074"
Sep 16 13:09:31.420232 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:31.420194    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1ef95c/auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft"]
Sep 16 13:09:31.485177 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:31.485147    2577 generic.go:356] "Generic (PLEG): container finished" podID="4559413e-b78b-4499-b85a-81ad046f306f" containerID="1364073e0b10fa0908c6bc1c1b4b3390d567aa54f826e1df2029e4ee4ac6bfa3" exitCode=0
Sep 16 13:09:31.485544 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:31.485226    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e5926f-hfcpn" event={"ID":"4559413e-b78b-4499-b85a-81ad046f306f","Type":"ContainerDied","Data":"1364073e0b10fa0908c6bc1c1b4b3390d567aa54f826e1df2029e4ee4ac6bfa3"}
Sep 16 13:09:32.636986 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:32.636962    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e5926f-hfcpn"
Sep 16 13:09:32.798656 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:32.798622    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4559413e-b78b-4499-b85a-81ad046f306f-maas-api-service-ca\") pod \"4559413e-b78b-4499-b85a-81ad046f306f\" (UID: \"4559413e-b78b-4499-b85a-81ad046f306f\") "
Sep 16 13:09:32.798982 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:32.798960    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4559413e-b78b-4499-b85a-81ad046f306f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4559413e-b78b-4499-b85a-81ad046f306f" (UID: "4559413e-b78b-4499-b85a-81ad046f306f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:09:32.900482 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:32.900439    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4559413e-b78b-4499-b85a-81ad046f306f-maas-api-service-ca\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:32.905352 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:32.905325    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d0256-29826060-sjmst"]
Sep 16 13:09:32.908736 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:32.908710    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d0256-29826060-sjmst"]
Sep 16 13:09:33.154064 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:33.153964    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7d0256-kpksp"]
Sep 16 13:09:33.157786 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:33.157758    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7d0256-kpksp"]
Sep 16 13:09:33.501189 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:33.501114    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e5926f-hfcpn"
Sep 16 13:09:33.501189 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:33.501162    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e5926f-hfcpn" event={"ID":"4559413e-b78b-4499-b85a-81ad046f306f","Type":"ContainerDied","Data":"24e01e31986318b5a5b4e3d5803e9b0c4e8e1949306c05bfbce99f84ec4883aa"}
Sep 16 13:09:33.501379 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:33.501216    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24e01e31986318b5a5b4e3d5803e9b0c4e8e1949306c05bfbce99f84ec4883aa"
Sep 16 13:09:34.059850 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:34.059817    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ba05653-57c7-4e1d-8c93-c150188ff517" path="/var/lib/kubelet/pods/6ba05653-57c7-4e1d-8c93-c150188ff517/volumes"
Sep 16 13:09:34.060223 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:34.060174    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d749be5d-502a-47ee-8505-edcf86455e50" path="/var/lib/kubelet/pods/d749be5d-502a-47ee-8505-edcf86455e50/volumes"
Sep 16 13:09:37.002931 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.002843    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7d0256/distinct-w4-7d0256-kserve-556b55b77-xcvvp" podUID="7735aca7-7d6b-464d-872a-00a4709feaec" containerName="main" containerID="cri-o://0abc2760f53a02526e172e88dbf7d1372f0486058d9830cd51ce2cef97a8955c" gracePeriod=45
Sep 16 13:09:37.016271 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.016235    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7d0256/premium-w4-7d0256-kserve-69b8fcc96b-bcwl4" podUID="08e00b87-97af-456a-af30-10b145da3ea6" containerName="main" containerID="cri-o://8d77ab72c7fd0ea0d9eee9b530dc7d541793cf90f92e652f01344a86579a69ae" gracePeriod=45
Sep 16 13:09:37.282378 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.282349    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d0256/distinct-w4-7d0256-kserve-556b55b77-xcvvp"
Sep 16 13:09:37.320916 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.320885    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d0256/premium-w4-7d0256-kserve-69b8fcc96b-bcwl4"
Sep 16 13:09:37.443508 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.443478    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08e00b87-97af-456a-af30-10b145da3ea6-dshm\") pod \"08e00b87-97af-456a-af30-10b145da3ea6\" (UID: \"08e00b87-97af-456a-af30-10b145da3ea6\") "
Sep 16 13:09:37.443508 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.443517    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08e00b87-97af-456a-af30-10b145da3ea6-home\") pod \"08e00b87-97af-456a-af30-10b145da3ea6\" (UID: \"08e00b87-97af-456a-af30-10b145da3ea6\") "
Sep 16 13:09:37.443772 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.443558    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwkw6\" (UniqueName: \"kubernetes.io/projected/7735aca7-7d6b-464d-872a-00a4709feaec-kube-api-access-cwkw6\") pod \"7735aca7-7d6b-464d-872a-00a4709feaec\" (UID: \"7735aca7-7d6b-464d-872a-00a4709feaec\") "
Sep 16 13:09:37.443772 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.443580    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7735aca7-7d6b-464d-872a-00a4709feaec-dshm\") pod \"7735aca7-7d6b-464d-872a-00a4709feaec\" (UID: \"7735aca7-7d6b-464d-872a-00a4709feaec\") "
Sep 16 13:09:37.443772 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.443612    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7735aca7-7d6b-464d-872a-00a4709feaec-tls-certs\") pod \"7735aca7-7d6b-464d-872a-00a4709feaec\" (UID: \"7735aca7-7d6b-464d-872a-00a4709feaec\") "
Sep 16 13:09:37.443772 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.443649    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08e00b87-97af-456a-af30-10b145da3ea6-model-cache\") pod \"08e00b87-97af-456a-af30-10b145da3ea6\" (UID: \"08e00b87-97af-456a-af30-10b145da3ea6\") "
Sep 16 13:09:37.443772 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.443692    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7735aca7-7d6b-464d-872a-00a4709feaec-tmp-dir\") pod \"7735aca7-7d6b-464d-872a-00a4709feaec\" (UID: \"7735aca7-7d6b-464d-872a-00a4709feaec\") "
Sep 16 13:09:37.443772 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.443743    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08e00b87-97af-456a-af30-10b145da3ea6-tmp-dir\") pod \"08e00b87-97af-456a-af30-10b145da3ea6\" (UID: \"08e00b87-97af-456a-af30-10b145da3ea6\") "
Sep 16 13:09:37.444142 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.443795    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08e00b87-97af-456a-af30-10b145da3ea6-home" (OuterVolumeSpecName: "home") pod "08e00b87-97af-456a-af30-10b145da3ea6" (UID: "08e00b87-97af-456a-af30-10b145da3ea6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:37.444142 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.443817    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7735aca7-7d6b-464d-872a-00a4709feaec-home\") pod \"7735aca7-7d6b-464d-872a-00a4709feaec\" (UID: \"7735aca7-7d6b-464d-872a-00a4709feaec\") "
Sep 16 13:09:37.444142 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.443871    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmp5s\" (UniqueName: \"kubernetes.io/projected/08e00b87-97af-456a-af30-10b145da3ea6-kube-api-access-rmp5s\") pod \"08e00b87-97af-456a-af30-10b145da3ea6\" (UID: \"08e00b87-97af-456a-af30-10b145da3ea6\") "
Sep 16 13:09:37.444142 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.443904    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7735aca7-7d6b-464d-872a-00a4709feaec-model-cache\") pod \"7735aca7-7d6b-464d-872a-00a4709feaec\" (UID: \"7735aca7-7d6b-464d-872a-00a4709feaec\") "
Sep 16 13:09:37.444142 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.443958    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7735aca7-7d6b-464d-872a-00a4709feaec-home" (OuterVolumeSpecName: "home") pod "7735aca7-7d6b-464d-872a-00a4709feaec" (UID: "7735aca7-7d6b-464d-872a-00a4709feaec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:37.444142 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.443975    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08e00b87-97af-456a-af30-10b145da3ea6-tls-certs\") pod \"08e00b87-97af-456a-af30-10b145da3ea6\" (UID: \"08e00b87-97af-456a-af30-10b145da3ea6\") "
Sep 16 13:09:37.444465 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.444322    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7735aca7-7d6b-464d-872a-00a4709feaec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7735aca7-7d6b-464d-872a-00a4709feaec" (UID: "7735aca7-7d6b-464d-872a-00a4709feaec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:37.444465 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.444355    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08e00b87-97af-456a-af30-10b145da3ea6-model-cache" (OuterVolumeSpecName: "model-cache") pod "08e00b87-97af-456a-af30-10b145da3ea6" (UID: "08e00b87-97af-456a-af30-10b145da3ea6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:37.444465 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.444428    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08e00b87-97af-456a-af30-10b145da3ea6-model-cache\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:37.444465 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.444448    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7735aca7-7d6b-464d-872a-00a4709feaec-tmp-dir\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:37.444465 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.444451    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08e00b87-97af-456a-af30-10b145da3ea6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "08e00b87-97af-456a-af30-10b145da3ea6" (UID: "08e00b87-97af-456a-af30-10b145da3ea6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:37.444465 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.444462    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7735aca7-7d6b-464d-872a-00a4709feaec-home\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:37.444788 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.444475    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08e00b87-97af-456a-af30-10b145da3ea6-home\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:37.444788 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.444593    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7735aca7-7d6b-464d-872a-00a4709feaec-model-cache" (OuterVolumeSpecName: "model-cache") pod "7735aca7-7d6b-464d-872a-00a4709feaec" (UID: "7735aca7-7d6b-464d-872a-00a4709feaec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:37.446595 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.446556    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08e00b87-97af-456a-af30-10b145da3ea6-dshm" (OuterVolumeSpecName: "dshm") pod "08e00b87-97af-456a-af30-10b145da3ea6" (UID: "08e00b87-97af-456a-af30-10b145da3ea6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:37.447841 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.447796    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7735aca7-7d6b-464d-872a-00a4709feaec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7735aca7-7d6b-464d-872a-00a4709feaec" (UID: "7735aca7-7d6b-464d-872a-00a4709feaec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:09:37.448524 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.448478    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08e00b87-97af-456a-af30-10b145da3ea6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "08e00b87-97af-456a-af30-10b145da3ea6" (UID: "08e00b87-97af-456a-af30-10b145da3ea6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:09:37.448654 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.448638    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08e00b87-97af-456a-af30-10b145da3ea6-kube-api-access-rmp5s" (OuterVolumeSpecName: "kube-api-access-rmp5s") pod "08e00b87-97af-456a-af30-10b145da3ea6" (UID: "08e00b87-97af-456a-af30-10b145da3ea6"). InnerVolumeSpecName "kube-api-access-rmp5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:09:37.449881 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.449850    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7735aca7-7d6b-464d-872a-00a4709feaec-kube-api-access-cwkw6" (OuterVolumeSpecName: "kube-api-access-cwkw6") pod "7735aca7-7d6b-464d-872a-00a4709feaec" (UID: "7735aca7-7d6b-464d-872a-00a4709feaec"). InnerVolumeSpecName "kube-api-access-cwkw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:09:37.450651 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.450622    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7735aca7-7d6b-464d-872a-00a4709feaec-dshm" (OuterVolumeSpecName: "dshm") pod "7735aca7-7d6b-464d-872a-00a4709feaec" (UID: "7735aca7-7d6b-464d-872a-00a4709feaec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:37.528875 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.528775    2577 generic.go:356] "Generic (PLEG): container finished" podID="08e00b87-97af-456a-af30-10b145da3ea6" containerID="8d77ab72c7fd0ea0d9eee9b530dc7d541793cf90f92e652f01344a86579a69ae" exitCode=0
Sep 16 13:09:37.528875 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.528846    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d0256/premium-w4-7d0256-kserve-69b8fcc96b-bcwl4"
Sep 16 13:09:37.529116 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.528875    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d0256/premium-w4-7d0256-kserve-69b8fcc96b-bcwl4" event={"ID":"08e00b87-97af-456a-af30-10b145da3ea6","Type":"ContainerDied","Data":"8d77ab72c7fd0ea0d9eee9b530dc7d541793cf90f92e652f01344a86579a69ae"}
Sep 16 13:09:37.529116 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.528927    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d0256/premium-w4-7d0256-kserve-69b8fcc96b-bcwl4" event={"ID":"08e00b87-97af-456a-af30-10b145da3ea6","Type":"ContainerDied","Data":"f23400508fde230740972f05a656ad6fd1193d06c4838450e2c1a4cfc688b48f"}
Sep 16 13:09:37.529116 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.528950    2577 scope.go:117] "RemoveContainer" containerID="8d77ab72c7fd0ea0d9eee9b530dc7d541793cf90f92e652f01344a86579a69ae"
Sep 16 13:09:37.530905 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.530880    2577 generic.go:356] "Generic (PLEG): container finished" podID="7735aca7-7d6b-464d-872a-00a4709feaec" containerID="0abc2760f53a02526e172e88dbf7d1372f0486058d9830cd51ce2cef97a8955c" exitCode=0
Sep 16 13:09:37.531003 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.530987    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d0256/distinct-w4-7d0256-kserve-556b55b77-xcvvp" event={"ID":"7735aca7-7d6b-464d-872a-00a4709feaec","Type":"ContainerDied","Data":"0abc2760f53a02526e172e88dbf7d1372f0486058d9830cd51ce2cef97a8955c"}
Sep 16 13:09:37.531109 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.531016    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d0256/distinct-w4-7d0256-kserve-556b55b77-xcvvp" event={"ID":"7735aca7-7d6b-464d-872a-00a4709feaec","Type":"ContainerDied","Data":"0ff3b33e27259e7ccc8b41cf5b35bfa19d946cb9db0a03259b6f6d94234ef208"}
Sep 16 13:09:37.531173 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.531113    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d0256/distinct-w4-7d0256-kserve-556b55b77-xcvvp"
Sep 16 13:09:37.539175 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.539153    2577 scope.go:117] "RemoveContainer" containerID="8d77ab72c7fd0ea0d9eee9b530dc7d541793cf90f92e652f01344a86579a69ae"
Sep 16 13:09:37.539733 ip-10-0-133-190 kubenswrapper[2577]: E0916 13:09:37.539494    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d77ab72c7fd0ea0d9eee9b530dc7d541793cf90f92e652f01344a86579a69ae\": container with ID starting with 8d77ab72c7fd0ea0d9eee9b530dc7d541793cf90f92e652f01344a86579a69ae not found: ID does not exist" containerID="8d77ab72c7fd0ea0d9eee9b530dc7d541793cf90f92e652f01344a86579a69ae"
Sep 16 13:09:37.539733 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.539532    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d77ab72c7fd0ea0d9eee9b530dc7d541793cf90f92e652f01344a86579a69ae"} err="failed to get container status \"8d77ab72c7fd0ea0d9eee9b530dc7d541793cf90f92e652f01344a86579a69ae\": rpc error: code = NotFound desc = could not find container \"8d77ab72c7fd0ea0d9eee9b530dc7d541793cf90f92e652f01344a86579a69ae\": container with ID starting with 8d77ab72c7fd0ea0d9eee9b530dc7d541793cf90f92e652f01344a86579a69ae not found: ID does not exist"
Sep 16 13:09:37.539733 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.539557    2577 scope.go:117] "RemoveContainer" containerID="0abc2760f53a02526e172e88dbf7d1372f0486058d9830cd51ce2cef97a8955c"
Sep 16 13:09:37.544978 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.544959    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08e00b87-97af-456a-af30-10b145da3ea6-tls-certs\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:37.544978 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.544981    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08e00b87-97af-456a-af30-10b145da3ea6-dshm\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:37.545151 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.544995    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwkw6\" (UniqueName: \"kubernetes.io/projected/7735aca7-7d6b-464d-872a-00a4709feaec-kube-api-access-cwkw6\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:37.545151 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.545009    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7735aca7-7d6b-464d-872a-00a4709feaec-dshm\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:37.545151 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.545022    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7735aca7-7d6b-464d-872a-00a4709feaec-tls-certs\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:37.545151 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.545049    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08e00b87-97af-456a-af30-10b145da3ea6-tmp-dir\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:37.545151 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.545061    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmp5s\" (UniqueName: \"kubernetes.io/projected/08e00b87-97af-456a-af30-10b145da3ea6-kube-api-access-rmp5s\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:37.545151 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.545073    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7735aca7-7d6b-464d-872a-00a4709feaec-model-cache\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:37.553529 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.552112    2577 scope.go:117] "RemoveContainer" containerID="0abc2760f53a02526e172e88dbf7d1372f0486058d9830cd51ce2cef97a8955c"
Sep 16 13:09:37.555067 ip-10-0-133-190 kubenswrapper[2577]: E0916 13:09:37.553647    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0abc2760f53a02526e172e88dbf7d1372f0486058d9830cd51ce2cef97a8955c\": container with ID starting with 0abc2760f53a02526e172e88dbf7d1372f0486058d9830cd51ce2cef97a8955c not found: ID does not exist" containerID="0abc2760f53a02526e172e88dbf7d1372f0486058d9830cd51ce2cef97a8955c"
Sep 16 13:09:37.555067 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.553686    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0abc2760f53a02526e172e88dbf7d1372f0486058d9830cd51ce2cef97a8955c"} err="failed to get container status \"0abc2760f53a02526e172e88dbf7d1372f0486058d9830cd51ce2cef97a8955c\": rpc error: code = NotFound desc = could not find container \"0abc2760f53a02526e172e88dbf7d1372f0486058d9830cd51ce2cef97a8955c\": container with ID starting with 0abc2760f53a02526e172e88dbf7d1372f0486058d9830cd51ce2cef97a8955c not found: ID does not exist"
Sep 16 13:09:37.561827 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.561801    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7d0256/distinct-w4-7d0256-kserve-556b55b77-xcvvp"]
Sep 16 13:09:37.564332 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.564310    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7d0256/distinct-w4-7d0256-kserve-556b55b77-xcvvp"]
Sep 16 13:09:37.575491 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.575468    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7d0256/premium-w4-7d0256-kserve-69b8fcc96b-bcwl4"]
Sep 16 13:09:37.581441 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:37.581421    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7d0256/premium-w4-7d0256-kserve-69b8fcc96b-bcwl4"]
Sep 16 13:09:38.056203 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:38.056172    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08e00b87-97af-456a-af30-10b145da3ea6" path="/var/lib/kubelet/pods/08e00b87-97af-456a-af30-10b145da3ea6/volumes"
Sep 16 13:09:38.056569 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:38.056555    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7735aca7-7d6b-464d-872a-00a4709feaec" path="/var/lib/kubelet/pods/7735aca7-7d6b-464d-872a-00a4709feaec/volumes"
Sep 16 13:09:39.667849 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:39.667816    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"]
Sep 16 13:09:39.668371 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:39.668048    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz" podUID="99b70637-81ce-4503-bc3d-de5dc69e7b97" containerName="istio-proxy" containerID="cri-o://dc0cf4a0f8fa83db54422e662098bda0c4142e1a1339200e0e025e018eae94f7" gracePeriod=30
Sep 16 13:09:39.921623 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:39.921542    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e5926f-7bcd585465-ms5wx"]
Sep 16 13:09:39.921816 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:39.921779    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e5926f-7bcd585465-ms5wx" podUID="01a3c352-0789-42df-8d81-3242438a84c2" containerName="payload-processing" containerID="cri-o://2d037a63782a56d1c0f3e55a68e666005b5d81620d517323f30c7f4d1eafd27a" gracePeriod=30
Sep 16 13:09:39.930215 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:39.930088    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5926f-68b6695bbb-cdq2j"]
Sep 16 13:09:39.930349 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:39.930308    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5926f-68b6695bbb-cdq2j" podUID="01d14db3-8e1d-4462-a9a6-f92962543f90" containerName="payload-pre-processing" containerID="cri-o://e50d7008229a715498e2295e2c5bf3f8dc0d13e9f0b2511a5e0b6e37ff94e900" gracePeriod=30
Sep 16 13:09:40.086379 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:40.086335    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e5926f-hfcpn"]
Sep 16 13:09:40.089127 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:40.089089    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e5926f-hfcpn"]
Sep 16 13:09:40.207486 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:40.207457    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e5926f-7bcd585465-ms5wx"
Sep 16 13:09:40.222639 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:40.222620    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5926f-68b6695bbb-cdq2j"
Sep 16 13:09:40.273965 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:40.273930    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7mjl\" (UniqueName: \"kubernetes.io/projected/01a3c352-0789-42df-8d81-3242438a84c2-kube-api-access-l7mjl\") pod \"01a3c352-0789-42df-8d81-3242438a84c2\" (UID: \"01a3c352-0789-42df-8d81-3242438a84c2\") "
Sep 16 13:09:40.274195 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:40.273980    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01a3c352-0789-42df-8d81-3242438a84c2-plugins-config-volume\") pod \"01a3c352-0789-42df-8d81-3242438a84c2\" (UID: \"01a3c352-0789-42df-8d81-3242438a84c2\") "
Sep 16 13:09:40.274195 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:40.274128    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxn6q\" (UniqueName: \"kubernetes.io/projected/01d14db3-8e1d-4462-a9a6-f92962543f90-kube-api-access-nxn6q\") pod \"01d14db3-8e1d-4462-a9a6-f92962543f90\" (UID: \"01d14db3-8e1d-4462-a9a6-f92962543f90\") "
Sep 16 13:09:40.274322 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:40.274198    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d14db3-8e1d-4462-a9a6-f92962543f90-plugins-config-volume\") pod \"01d14db3-8e1d-4462-a9a6-f92962543f90\" (UID: \"01d14db3-8e1d-4462-a9a6-f92962543f90\") "
Sep 16 13:09:40.274474 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:40.274447    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01a3c352-0789-42df-8d81-3242438a84c2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "01a3c352-0789-42df-8d81-3242438a84c2" (UID: "01a3c352-0789-42df-8d81-3242438a84c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:09:40.274874 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:40.274801    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01d14db3-8e1d-4462-a9a6-f92962543f90-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "01d14db3-8e1d-4462-a9a6-f92962543f90" (UID: "01d14db3-8e1d-4462-a9a6-f92962543f90"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:09:40.276520 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:40.276497    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01d14db3-8e1d-4462-a9a6-f92962543f90-kube-api-access-nxn6q" (OuterVolumeSpecName: "kube-api-access-nxn6q") pod "01d14db3-8e1d-4462-a9a6-f92962543f90" (UID: "01d14db3-8e1d-4462-a9a6-f92962543f90"). InnerVolumeSpecName "kube-api-access-nxn6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:09:40.276614 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:40.276573    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a3c352-0789-42df-8d81-3242438a84c2-kube-api-access-l7mjl" (OuterVolumeSpecName: "kube-api-access-l7mjl") pod "01a3c352-0789-42df-8d81-3242438a84c2" (UID: "01a3c352-0789-42df-8d81-3242438a84c2"). InnerVolumeSpecName "kube-api-access-l7mjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:09:40.375138 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:40.375106    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxn6q\" (UniqueName: \"kubernetes.io/projected/01d14db3-8e1d-4462-a9a6-f92962543f90-kube-api-access-nxn6q\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:40.375138 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:40.375134    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d14db3-8e1d-4462-a9a6-f92962543f90-plugins-config-volume\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:40.375138 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:40.375144    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7mjl\" (UniqueName: \"kubernetes.io/projected/01a3c352-0789-42df-8d81-3242438a84c2-kube-api-access-l7mjl\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:40.375368 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:40.375153    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01a3c352-0789-42df-8d81-3242438a84c2-plugins-config-volume\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:40.553433 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:40.553396    2577 generic.go:356] "Generic (PLEG): container finished" podID="01d14db3-8e1d-4462-a9a6-f92962543f90" containerID="e50d7008229a715498e2295e2c5bf3f8dc0d13e9f0b2511a5e0b6e37ff94e900" exitCode=0
Sep 16 13:09:40.553607 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:40.553462    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5926f-68b6695bbb-cdq2j"
Sep 16 13:09:40.553607 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:40.553482    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5926f-68b6695bbb-cdq2j" event={"ID":"01d14db3-8e1d-4462-a9a6-f92962543f90","Type":"ContainerDied","Data":"e50d7008229a715498e2295e2c5bf3f8dc0d13e9f0b2511a5e0b6e37ff94e900"}
Sep 16 13:09:40.553607 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:40.553541    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5926f-68b6695bbb-cdq2j" event={"ID":"01d14db3-8e1d-4462-a9a6-f92962543f90","Type":"ContainerDied","Data":"740bdcdd1af8e7b0e8cdffa42e16c869e666b459a83f7e25df4b776fb0d094e5"}
Sep 16 13:09:40.553607 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:40.553565    2577 scope.go:117] "RemoveContainer" containerID="e50d7008229a715498e2295e2c5bf3f8dc0d13e9f0b2511a5e0b6e37ff94e900"
Sep 16 13:09:40.554952 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:40.554932    2577 generic.go:356] "Generic (PLEG): container finished" podID="01a3c352-0789-42df-8d81-3242438a84c2" containerID="2d037a63782a56d1c0f3e55a68e666005b5d81620d517323f30c7f4d1eafd27a" exitCode=0
Sep 16 13:09:40.555084 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:40.554974    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e5926f-7bcd585465-ms5wx"
Sep 16 13:09:40.555084 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:40.554978    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e5926f-7bcd585465-ms5wx" event={"ID":"01a3c352-0789-42df-8d81-3242438a84c2","Type":"ContainerDied","Data":"2d037a63782a56d1c0f3e55a68e666005b5d81620d517323f30c7f4d1eafd27a"}
Sep 16 13:09:40.555257 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:40.555092    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e5926f-7bcd585465-ms5wx" event={"ID":"01a3c352-0789-42df-8d81-3242438a84c2","Type":"ContainerDied","Data":"ce20471bf2620cdae1f8d57de2f60356ce1915df422f35dcf332bdbf9c943101"}
Sep 16 13:09:40.565014 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:40.564992    2577 scope.go:117] "RemoveContainer" containerID="e50d7008229a715498e2295e2c5bf3f8dc0d13e9f0b2511a5e0b6e37ff94e900"
Sep 16 13:09:40.565441 ip-10-0-133-190 kubenswrapper[2577]: E0916 13:09:40.565416    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e50d7008229a715498e2295e2c5bf3f8dc0d13e9f0b2511a5e0b6e37ff94e900\": container with ID starting with e50d7008229a715498e2295e2c5bf3f8dc0d13e9f0b2511a5e0b6e37ff94e900 not found: ID does not exist" containerID="e50d7008229a715498e2295e2c5bf3f8dc0d13e9f0b2511a5e0b6e37ff94e900"
Sep 16 13:09:40.565533 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:40.565445    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e50d7008229a715498e2295e2c5bf3f8dc0d13e9f0b2511a5e0b6e37ff94e900"} err="failed to get container status \"e50d7008229a715498e2295e2c5bf3f8dc0d13e9f0b2511a5e0b6e37ff94e900\": rpc error: code = NotFound desc = could not find container \"e50d7008229a715498e2295e2c5bf3f8dc0d13e9f0b2511a5e0b6e37ff94e900\": container with ID starting with e50d7008229a715498e2295e2c5bf3f8dc0d13e9f0b2511a5e0b6e37ff94e900 not found: ID does not exist"
Sep 16 13:09:40.565533 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:40.565461    2577 scope.go:117] "RemoveContainer" containerID="2d037a63782a56d1c0f3e55a68e666005b5d81620d517323f30c7f4d1eafd27a"
Sep 16 13:09:40.574405 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:40.574377    2577 scope.go:117] "RemoveContainer" containerID="2d037a63782a56d1c0f3e55a68e666005b5d81620d517323f30c7f4d1eafd27a"
Sep 16 13:09:40.574662 ip-10-0-133-190 kubenswrapper[2577]: E0916 13:09:40.574641    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d037a63782a56d1c0f3e55a68e666005b5d81620d517323f30c7f4d1eafd27a\": container with ID starting with 2d037a63782a56d1c0f3e55a68e666005b5d81620d517323f30c7f4d1eafd27a not found: ID does not exist" containerID="2d037a63782a56d1c0f3e55a68e666005b5d81620d517323f30c7f4d1eafd27a"
Sep 16 13:09:40.574730 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:40.574672    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d037a63782a56d1c0f3e55a68e666005b5d81620d517323f30c7f4d1eafd27a"} err="failed to get container status \"2d037a63782a56d1c0f3e55a68e666005b5d81620d517323f30c7f4d1eafd27a\": rpc error: code = NotFound desc = could not find container \"2d037a63782a56d1c0f3e55a68e666005b5d81620d517323f30c7f4d1eafd27a\": container with ID starting with 2d037a63782a56d1c0f3e55a68e666005b5d81620d517323f30c7f4d1eafd27a not found: ID does not exist"
Sep 16 13:09:40.582453 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:40.582430    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5926f-68b6695bbb-cdq2j"]
Sep 16 13:09:40.586459 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:40.586435    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5926f-68b6695bbb-cdq2j"]
Sep 16 13:09:40.598039 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:40.598010    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e5926f-7bcd585465-ms5wx"]
Sep 16 13:09:40.604613 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:40.604587    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e5926f-7bcd585465-ms5wx"]
Sep 16 13:09:42.056037 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:42.055992    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01a3c352-0789-42df-8d81-3242438a84c2" path="/var/lib/kubelet/pods/01a3c352-0789-42df-8d81-3242438a84c2/volumes"
Sep 16 13:09:42.056418 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:42.056392    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01d14db3-8e1d-4462-a9a6-f92962543f90" path="/var/lib/kubelet/pods/01d14db3-8e1d-4462-a9a6-f92962543f90/volumes"
Sep 16 13:09:42.056680 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:42.056667    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4559413e-b78b-4499-b85a-81ad046f306f" path="/var/lib/kubelet/pods/4559413e-b78b-4499-b85a-81ad046f306f/volumes"
Sep 16 13:09:44.922669 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:44.922647    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"
Sep 16 13:09:45.019671 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:45.019642    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99b70637-81ce-4503-bc3d-de5dc69e7b97-istio-token\") pod \"99b70637-81ce-4503-bc3d-de5dc69e7b97\" (UID: \"99b70637-81ce-4503-bc3d-de5dc69e7b97\") "
Sep 16 13:09:45.019836 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:45.019681    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99b70637-81ce-4503-bc3d-de5dc69e7b97-istio-data\") pod \"99b70637-81ce-4503-bc3d-de5dc69e7b97\" (UID: \"99b70637-81ce-4503-bc3d-de5dc69e7b97\") "
Sep 16 13:09:45.019836 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:45.019721    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b70637-81ce-4503-bc3d-de5dc69e7b97-credential-socket\") pod \"99b70637-81ce-4503-bc3d-de5dc69e7b97\" (UID: \"99b70637-81ce-4503-bc3d-de5dc69e7b97\") "
Sep 16 13:09:45.019836 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:45.019741    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99b70637-81ce-4503-bc3d-de5dc69e7b97-istio-envoy\") pod \"99b70637-81ce-4503-bc3d-de5dc69e7b97\" (UID: \"99b70637-81ce-4503-bc3d-de5dc69e7b97\") "
Sep 16 13:09:45.019836 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:45.019763    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99b70637-81ce-4503-bc3d-de5dc69e7b97-workload-certs\") pod \"99b70637-81ce-4503-bc3d-de5dc69e7b97\" (UID: \"99b70637-81ce-4503-bc3d-de5dc69e7b97\") "
Sep 16 13:09:45.020088 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:45.019862    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbqpp\" (UniqueName: \"kubernetes.io/projected/99b70637-81ce-4503-bc3d-de5dc69e7b97-kube-api-access-qbqpp\") pod \"99b70637-81ce-4503-bc3d-de5dc69e7b97\" (UID: \"99b70637-81ce-4503-bc3d-de5dc69e7b97\") "
Sep 16 13:09:45.020088 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:45.019911    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99b70637-81ce-4503-bc3d-de5dc69e7b97-istiod-ca-cert\") pod \"99b70637-81ce-4503-bc3d-de5dc69e7b97\" (UID: \"99b70637-81ce-4503-bc3d-de5dc69e7b97\") "
Sep 16 13:09:45.020088 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:45.019949    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b70637-81ce-4503-bc3d-de5dc69e7b97-workload-socket\") pod \"99b70637-81ce-4503-bc3d-de5dc69e7b97\" (UID: \"99b70637-81ce-4503-bc3d-de5dc69e7b97\") "
Sep 16 13:09:45.020088 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:45.019978    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99b70637-81ce-4503-bc3d-de5dc69e7b97-istio-podinfo\") pod \"99b70637-81ce-4503-bc3d-de5dc69e7b97\" (UID: \"99b70637-81ce-4503-bc3d-de5dc69e7b97\") "
Sep 16 13:09:45.020088 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:45.020008    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b70637-81ce-4503-bc3d-de5dc69e7b97-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "99b70637-81ce-4503-bc3d-de5dc69e7b97" (UID: "99b70637-81ce-4503-bc3d-de5dc69e7b97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:45.020389 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:45.020091    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b70637-81ce-4503-bc3d-de5dc69e7b97-istio-data" (OuterVolumeSpecName: "istio-data") pod "99b70637-81ce-4503-bc3d-de5dc69e7b97" (UID: "99b70637-81ce-4503-bc3d-de5dc69e7b97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:45.020389 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:45.020187    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b70637-81ce-4503-bc3d-de5dc69e7b97-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "99b70637-81ce-4503-bc3d-de5dc69e7b97" (UID: "99b70637-81ce-4503-bc3d-de5dc69e7b97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:45.020389 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:45.020271    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b70637-81ce-4503-bc3d-de5dc69e7b97-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "99b70637-81ce-4503-bc3d-de5dc69e7b97" (UID: "99b70637-81ce-4503-bc3d-de5dc69e7b97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:45.020535 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:45.020425    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b70637-81ce-4503-bc3d-de5dc69e7b97-workload-socket\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:45.020535 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:45.020446    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99b70637-81ce-4503-bc3d-de5dc69e7b97-istio-data\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:45.020535 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:45.020462    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b70637-81ce-4503-bc3d-de5dc69e7b97-credential-socket\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:45.020535 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:45.020476    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99b70637-81ce-4503-bc3d-de5dc69e7b97-workload-certs\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:45.020692 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:45.020552    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99b70637-81ce-4503-bc3d-de5dc69e7b97-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "99b70637-81ce-4503-bc3d-de5dc69e7b97" (UID: "99b70637-81ce-4503-bc3d-de5dc69e7b97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:09:45.022088 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:45.022067    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b70637-81ce-4503-bc3d-de5dc69e7b97-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "99b70637-81ce-4503-bc3d-de5dc69e7b97" (UID: "99b70637-81ce-4503-bc3d-de5dc69e7b97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:45.022234 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:45.022214    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99b70637-81ce-4503-bc3d-de5dc69e7b97-kube-api-access-qbqpp" (OuterVolumeSpecName: "kube-api-access-qbqpp") pod "99b70637-81ce-4503-bc3d-de5dc69e7b97" (UID: "99b70637-81ce-4503-bc3d-de5dc69e7b97"). InnerVolumeSpecName "kube-api-access-qbqpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:09:45.022539 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:45.022515    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99b70637-81ce-4503-bc3d-de5dc69e7b97-istio-token" (OuterVolumeSpecName: "istio-token") pod "99b70637-81ce-4503-bc3d-de5dc69e7b97" (UID: "99b70637-81ce-4503-bc3d-de5dc69e7b97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:09:45.022632 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:45.022540    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99b70637-81ce-4503-bc3d-de5dc69e7b97-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "99b70637-81ce-4503-bc3d-de5dc69e7b97" (UID: "99b70637-81ce-4503-bc3d-de5dc69e7b97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 13:09:45.121877 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:45.121846    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbqpp\" (UniqueName: \"kubernetes.io/projected/99b70637-81ce-4503-bc3d-de5dc69e7b97-kube-api-access-qbqpp\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:45.121877 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:45.121876    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99b70637-81ce-4503-bc3d-de5dc69e7b97-istiod-ca-cert\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:45.122092 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:45.121889    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99b70637-81ce-4503-bc3d-de5dc69e7b97-istio-podinfo\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:45.122092 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:45.121897    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99b70637-81ce-4503-bc3d-de5dc69e7b97-istio-token\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:45.122092 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:45.121905    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99b70637-81ce-4503-bc3d-de5dc69e7b97-istio-envoy\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:45.591449 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:45.591411    2577 generic.go:356] "Generic (PLEG): container finished" podID="99b70637-81ce-4503-bc3d-de5dc69e7b97" containerID="dc0cf4a0f8fa83db54422e662098bda0c4142e1a1339200e0e025e018eae94f7" exitCode=0
Sep 16 13:09:45.591678 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:45.591475    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"
Sep 16 13:09:45.591678 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:45.591489    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz" event={"ID":"99b70637-81ce-4503-bc3d-de5dc69e7b97","Type":"ContainerDied","Data":"dc0cf4a0f8fa83db54422e662098bda0c4142e1a1339200e0e025e018eae94f7"}
Sep 16 13:09:45.591678 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:45.591561    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz" event={"ID":"99b70637-81ce-4503-bc3d-de5dc69e7b97","Type":"ContainerDied","Data":"2ac85d1a96678df1933082b4c4941afccd308594341802c5a8944bf5442e62e5"}
Sep 16 13:09:45.591678 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:45.591591    2577 scope.go:117] "RemoveContainer" containerID="dc0cf4a0f8fa83db54422e662098bda0c4142e1a1339200e0e025e018eae94f7"
Sep 16 13:09:45.607669 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:45.607652    2577 scope.go:117] "RemoveContainer" containerID="472837c031edae40d3d5c05ad30b7ed63e00da96d92d905048f17581c59547d8"
Sep 16 13:09:45.616675 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:45.616656    2577 scope.go:117] "RemoveContainer" containerID="dc0cf4a0f8fa83db54422e662098bda0c4142e1a1339200e0e025e018eae94f7"
Sep 16 13:09:45.616940 ip-10-0-133-190 kubenswrapper[2577]: E0916 13:09:45.616921    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc0cf4a0f8fa83db54422e662098bda0c4142e1a1339200e0e025e018eae94f7\": container with ID starting with dc0cf4a0f8fa83db54422e662098bda0c4142e1a1339200e0e025e018eae94f7 not found: ID does not exist" containerID="dc0cf4a0f8fa83db54422e662098bda0c4142e1a1339200e0e025e018eae94f7"
Sep 16 13:09:45.616989 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:45.616949    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc0cf4a0f8fa83db54422e662098bda0c4142e1a1339200e0e025e018eae94f7"} err="failed to get container status \"dc0cf4a0f8fa83db54422e662098bda0c4142e1a1339200e0e025e018eae94f7\": rpc error: code = NotFound desc = could not find container \"dc0cf4a0f8fa83db54422e662098bda0c4142e1a1339200e0e025e018eae94f7\": container with ID starting with dc0cf4a0f8fa83db54422e662098bda0c4142e1a1339200e0e025e018eae94f7 not found: ID does not exist"
Sep 16 13:09:45.616989 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:45.616966    2577 scope.go:117] "RemoveContainer" containerID="472837c031edae40d3d5c05ad30b7ed63e00da96d92d905048f17581c59547d8"
Sep 16 13:09:45.617265 ip-10-0-133-190 kubenswrapper[2577]: E0916 13:09:45.617248    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"472837c031edae40d3d5c05ad30b7ed63e00da96d92d905048f17581c59547d8\": container with ID starting with 472837c031edae40d3d5c05ad30b7ed63e00da96d92d905048f17581c59547d8 not found: ID does not exist" containerID="472837c031edae40d3d5c05ad30b7ed63e00da96d92d905048f17581c59547d8"
Sep 16 13:09:45.617304 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:45.617272    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"472837c031edae40d3d5c05ad30b7ed63e00da96d92d905048f17581c59547d8"} err="failed to get container status \"472837c031edae40d3d5c05ad30b7ed63e00da96d92d905048f17581c59547d8\": rpc error: code = NotFound desc = could not find container \"472837c031edae40d3d5c05ad30b7ed63e00da96d92d905048f17581c59547d8\": container with ID starting with 472837c031edae40d3d5c05ad30b7ed63e00da96d92d905048f17581c59547d8 not found: ID does not exist"
Sep 16 13:09:45.633521 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:45.633414    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"]
Sep 16 13:09:45.635433 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:45.635415    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d0256-openshift-default-56f54765dd-m7wlz"]
Sep 16 13:09:46.055575 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:46.055533    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99b70637-81ce-4503-bc3d-de5dc69e7b97" path="/var/lib/kubelet/pods/99b70637-81ce-4503-bc3d-de5dc69e7b97/volumes"
Sep 16 13:09:46.441150 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:46.441060    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-1ef95c/auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft" podUID="6ddfa698-b491-497a-be0c-602cb4c1f8ed" containerName="main" containerID="cri-o://798fd6dd202f00e32f384a02fc6d9b852d70e07888ad52ed27de00c4c34e5b31" gracePeriod=45
Sep 16 13:09:46.600627 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:46.600590    2577 generic.go:356] "Generic (PLEG): container finished" podID="6ddfa698-b491-497a-be0c-602cb4c1f8ed" containerID="798fd6dd202f00e32f384a02fc6d9b852d70e07888ad52ed27de00c4c34e5b31" exitCode=0
Sep 16 13:09:46.600783 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:46.600661    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1ef95c/auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft" event={"ID":"6ddfa698-b491-497a-be0c-602cb4c1f8ed","Type":"ContainerDied","Data":"798fd6dd202f00e32f384a02fc6d9b852d70e07888ad52ed27de00c4c34e5b31"}
Sep 16 13:09:46.694870 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:46.694813    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1ef95c/auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft"
Sep 16 13:09:46.736347 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:46.736314    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ddfa698-b491-497a-be0c-602cb4c1f8ed-home\") pod \"6ddfa698-b491-497a-be0c-602cb4c1f8ed\" (UID: \"6ddfa698-b491-497a-be0c-602cb4c1f8ed\") "
Sep 16 13:09:46.736524 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:46.736366    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ddfa698-b491-497a-be0c-602cb4c1f8ed-model-cache\") pod \"6ddfa698-b491-497a-be0c-602cb4c1f8ed\" (UID: \"6ddfa698-b491-497a-be0c-602cb4c1f8ed\") "
Sep 16 13:09:46.736524 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:46.736446    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ddfa698-b491-497a-be0c-602cb4c1f8ed-tls-certs\") pod \"6ddfa698-b491-497a-be0c-602cb4c1f8ed\" (UID: \"6ddfa698-b491-497a-be0c-602cb4c1f8ed\") "
Sep 16 13:09:46.736524 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:46.736476    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jppj4\" (UniqueName: \"kubernetes.io/projected/6ddfa698-b491-497a-be0c-602cb4c1f8ed-kube-api-access-jppj4\") pod \"6ddfa698-b491-497a-be0c-602cb4c1f8ed\" (UID: \"6ddfa698-b491-497a-be0c-602cb4c1f8ed\") "
Sep 16 13:09:46.736524 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:46.736500    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ddfa698-b491-497a-be0c-602cb4c1f8ed-tmp-dir\") pod \"6ddfa698-b491-497a-be0c-602cb4c1f8ed\" (UID: \"6ddfa698-b491-497a-be0c-602cb4c1f8ed\") "
Sep 16 13:09:46.736731 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:46.736563    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ddfa698-b491-497a-be0c-602cb4c1f8ed-dshm\") pod \"6ddfa698-b491-497a-be0c-602cb4c1f8ed\" (UID: \"6ddfa698-b491-497a-be0c-602cb4c1f8ed\") "
Sep 16 13:09:46.737217 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:46.737083    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ddfa698-b491-497a-be0c-602cb4c1f8ed-model-cache" (OuterVolumeSpecName: "model-cache") pod "6ddfa698-b491-497a-be0c-602cb4c1f8ed" (UID: "6ddfa698-b491-497a-be0c-602cb4c1f8ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:46.737217 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:46.737096    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ddfa698-b491-497a-be0c-602cb4c1f8ed-home" (OuterVolumeSpecName: "home") pod "6ddfa698-b491-497a-be0c-602cb4c1f8ed" (UID: "6ddfa698-b491-497a-be0c-602cb4c1f8ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:46.737217 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:46.737140    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ddfa698-b491-497a-be0c-602cb4c1f8ed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6ddfa698-b491-497a-be0c-602cb4c1f8ed" (UID: "6ddfa698-b491-497a-be0c-602cb4c1f8ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:46.739647 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:46.739615    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ddfa698-b491-497a-be0c-602cb4c1f8ed-dshm" (OuterVolumeSpecName: "dshm") pod "6ddfa698-b491-497a-be0c-602cb4c1f8ed" (UID: "6ddfa698-b491-497a-be0c-602cb4c1f8ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:46.739777 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:46.739720    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ddfa698-b491-497a-be0c-602cb4c1f8ed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ddfa698-b491-497a-be0c-602cb4c1f8ed" (UID: "6ddfa698-b491-497a-be0c-602cb4c1f8ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:09:46.739777 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:46.739741    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ddfa698-b491-497a-be0c-602cb4c1f8ed-kube-api-access-jppj4" (OuterVolumeSpecName: "kube-api-access-jppj4") pod "6ddfa698-b491-497a-be0c-602cb4c1f8ed" (UID: "6ddfa698-b491-497a-be0c-602cb4c1f8ed"). InnerVolumeSpecName "kube-api-access-jppj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:09:46.837549 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:46.837513    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ddfa698-b491-497a-be0c-602cb4c1f8ed-tls-certs\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:46.837549 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:46.837543    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jppj4\" (UniqueName: \"kubernetes.io/projected/6ddfa698-b491-497a-be0c-602cb4c1f8ed-kube-api-access-jppj4\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:46.837549 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:46.837553    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ddfa698-b491-497a-be0c-602cb4c1f8ed-tmp-dir\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:46.837761 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:46.837565    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ddfa698-b491-497a-be0c-602cb4c1f8ed-dshm\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:46.837761 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:46.837573    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ddfa698-b491-497a-be0c-602cb4c1f8ed-home\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:46.837761 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:46.837580    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ddfa698-b491-497a-be0c-602cb4c1f8ed-model-cache\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:47.246317 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:47.246281    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"]
Sep 16 13:09:47.246785 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:47.246491    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn" podUID="05103a7a-76fc-4417-a62a-2d1439edc444" containerName="istio-proxy" containerID="cri-o://4ad2bf2790cbf968d8fd8a69b5861c34e0666a2749c56bcebbbe98c6bb7478a2" gracePeriod=30
Sep 16 13:09:47.609140 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:47.609109    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1ef95c/auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft"
Sep 16 13:09:47.609140 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:47.609131    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1ef95c/auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft" event={"ID":"6ddfa698-b491-497a-be0c-602cb4c1f8ed","Type":"ContainerDied","Data":"f6c102ea1258ef73f68498ee0271d2310698c56c9b1a543e8501155bfaeda7de"}
Sep 16 13:09:47.609406 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:47.609177    2577 scope.go:117] "RemoveContainer" containerID="798fd6dd202f00e32f384a02fc6d9b852d70e07888ad52ed27de00c4c34e5b31"
Sep 16 13:09:47.632346 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:47.632321    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1ef95c/auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft"]
Sep 16 13:09:47.636282 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:47.636258    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1ef95c/auth-test-model-1ef95c-kserve-55c5c4c9cf-dgfft"]
Sep 16 13:09:48.055218 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:48.055187    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ddfa698-b491-497a-be0c-602cb4c1f8ed" path="/var/lib/kubelet/pods/6ddfa698-b491-497a-be0c-602cb4c1f8ed/volumes"
Sep 16 13:09:52.500146 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.500120    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"
Sep 16 13:09:52.597415 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.597340    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7ffw\" (UniqueName: \"kubernetes.io/projected/05103a7a-76fc-4417-a62a-2d1439edc444-kube-api-access-r7ffw\") pod \"05103a7a-76fc-4417-a62a-2d1439edc444\" (UID: \"05103a7a-76fc-4417-a62a-2d1439edc444\") "
Sep 16 13:09:52.597415 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.597405    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05103a7a-76fc-4417-a62a-2d1439edc444-istio-envoy\") pod \"05103a7a-76fc-4417-a62a-2d1439edc444\" (UID: \"05103a7a-76fc-4417-a62a-2d1439edc444\") "
Sep 16 13:09:52.597636 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.597449    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05103a7a-76fc-4417-a62a-2d1439edc444-istiod-ca-cert\") pod \"05103a7a-76fc-4417-a62a-2d1439edc444\" (UID: \"05103a7a-76fc-4417-a62a-2d1439edc444\") "
Sep 16 13:09:52.597636 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.597477    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05103a7a-76fc-4417-a62a-2d1439edc444-workload-certs\") pod \"05103a7a-76fc-4417-a62a-2d1439edc444\" (UID: \"05103a7a-76fc-4417-a62a-2d1439edc444\") "
Sep 16 13:09:52.597636 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.597502    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05103a7a-76fc-4417-a62a-2d1439edc444-credential-socket\") pod \"05103a7a-76fc-4417-a62a-2d1439edc444\" (UID: \"05103a7a-76fc-4417-a62a-2d1439edc444\") "
Sep 16 13:09:52.597636 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.597537    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05103a7a-76fc-4417-a62a-2d1439edc444-istio-podinfo\") pod \"05103a7a-76fc-4417-a62a-2d1439edc444\" (UID: \"05103a7a-76fc-4417-a62a-2d1439edc444\") "
Sep 16 13:09:52.597636 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.597569    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05103a7a-76fc-4417-a62a-2d1439edc444-istio-data\") pod \"05103a7a-76fc-4417-a62a-2d1439edc444\" (UID: \"05103a7a-76fc-4417-a62a-2d1439edc444\") "
Sep 16 13:09:52.597636 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.597598    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05103a7a-76fc-4417-a62a-2d1439edc444-workload-socket\") pod \"05103a7a-76fc-4417-a62a-2d1439edc444\" (UID: \"05103a7a-76fc-4417-a62a-2d1439edc444\") "
Sep 16 13:09:52.597939 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.597872    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05103a7a-76fc-4417-a62a-2d1439edc444-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "05103a7a-76fc-4417-a62a-2d1439edc444" (UID: "05103a7a-76fc-4417-a62a-2d1439edc444"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:52.597939 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.597887    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05103a7a-76fc-4417-a62a-2d1439edc444-istio-data" (OuterVolumeSpecName: "istio-data") pod "05103a7a-76fc-4417-a62a-2d1439edc444" (UID: "05103a7a-76fc-4417-a62a-2d1439edc444"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:52.597939 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.597904    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05103a7a-76fc-4417-a62a-2d1439edc444-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "05103a7a-76fc-4417-a62a-2d1439edc444" (UID: "05103a7a-76fc-4417-a62a-2d1439edc444"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:52.598135 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.598066    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05103a7a-76fc-4417-a62a-2d1439edc444-istio-token\") pod \"05103a7a-76fc-4417-a62a-2d1439edc444\" (UID: \"05103a7a-76fc-4417-a62a-2d1439edc444\") "
Sep 16 13:09:52.598135 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.598117    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05103a7a-76fc-4417-a62a-2d1439edc444-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "05103a7a-76fc-4417-a62a-2d1439edc444" (UID: "05103a7a-76fc-4417-a62a-2d1439edc444"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:52.598229 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.598167    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05103a7a-76fc-4417-a62a-2d1439edc444-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "05103a7a-76fc-4417-a62a-2d1439edc444" (UID: "05103a7a-76fc-4417-a62a-2d1439edc444"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:09:52.598426 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.598409    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05103a7a-76fc-4417-a62a-2d1439edc444-istiod-ca-cert\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:52.598494 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.598432    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05103a7a-76fc-4417-a62a-2d1439edc444-workload-certs\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:52.598494 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.598445    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05103a7a-76fc-4417-a62a-2d1439edc444-credential-socket\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:52.598494 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.598461    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05103a7a-76fc-4417-a62a-2d1439edc444-istio-data\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:52.598494 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.598474    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05103a7a-76fc-4417-a62a-2d1439edc444-workload-socket\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:52.599799 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.599775    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05103a7a-76fc-4417-a62a-2d1439edc444-kube-api-access-r7ffw" (OuterVolumeSpecName: "kube-api-access-r7ffw") pod "05103a7a-76fc-4417-a62a-2d1439edc444" (UID: "05103a7a-76fc-4417-a62a-2d1439edc444"). InnerVolumeSpecName "kube-api-access-r7ffw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:09:52.600381 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.600351    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05103a7a-76fc-4417-a62a-2d1439edc444-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "05103a7a-76fc-4417-a62a-2d1439edc444" (UID: "05103a7a-76fc-4417-a62a-2d1439edc444"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:52.600381 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.600367    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05103a7a-76fc-4417-a62a-2d1439edc444-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "05103a7a-76fc-4417-a62a-2d1439edc444" (UID: "05103a7a-76fc-4417-a62a-2d1439edc444"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 13:09:52.600507 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.600414    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05103a7a-76fc-4417-a62a-2d1439edc444-istio-token" (OuterVolumeSpecName: "istio-token") pod "05103a7a-76fc-4417-a62a-2d1439edc444" (UID: "05103a7a-76fc-4417-a62a-2d1439edc444"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:09:52.643311 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.643280    2577 generic.go:356] "Generic (PLEG): container finished" podID="05103a7a-76fc-4417-a62a-2d1439edc444" containerID="4ad2bf2790cbf968d8fd8a69b5861c34e0666a2749c56bcebbbe98c6bb7478a2" exitCode=0
Sep 16 13:09:52.643487 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.643332    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"
Sep 16 13:09:52.643487 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.643367    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn" event={"ID":"05103a7a-76fc-4417-a62a-2d1439edc444","Type":"ContainerDied","Data":"4ad2bf2790cbf968d8fd8a69b5861c34e0666a2749c56bcebbbe98c6bb7478a2"}
Sep 16 13:09:52.643487 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.643435    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn" event={"ID":"05103a7a-76fc-4417-a62a-2d1439edc444","Type":"ContainerDied","Data":"4f784af204d06c73f46387627c8f80011f2ee21c567be3d930119fbb17f9f87c"}
Sep 16 13:09:52.643487 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.643464    2577 scope.go:117] "RemoveContainer" containerID="4ad2bf2790cbf968d8fd8a69b5861c34e0666a2749c56bcebbbe98c6bb7478a2"
Sep 16 13:09:52.654881 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.654862    2577 scope.go:117] "RemoveContainer" containerID="4597b6bb5db08ddf331310f64c8e1f1529b730ffd6466a285d45ef1b97e58640"
Sep 16 13:09:52.663561 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.663544    2577 scope.go:117] "RemoveContainer" containerID="4ad2bf2790cbf968d8fd8a69b5861c34e0666a2749c56bcebbbe98c6bb7478a2"
Sep 16 13:09:52.663836 ip-10-0-133-190 kubenswrapper[2577]: E0916 13:09:52.663816    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ad2bf2790cbf968d8fd8a69b5861c34e0666a2749c56bcebbbe98c6bb7478a2\": container with ID starting with 4ad2bf2790cbf968d8fd8a69b5861c34e0666a2749c56bcebbbe98c6bb7478a2 not found: ID does not exist" containerID="4ad2bf2790cbf968d8fd8a69b5861c34e0666a2749c56bcebbbe98c6bb7478a2"
Sep 16 13:09:52.663884 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.663846    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ad2bf2790cbf968d8fd8a69b5861c34e0666a2749c56bcebbbe98c6bb7478a2"} err="failed to get container status \"4ad2bf2790cbf968d8fd8a69b5861c34e0666a2749c56bcebbbe98c6bb7478a2\": rpc error: code = NotFound desc = could not find container \"4ad2bf2790cbf968d8fd8a69b5861c34e0666a2749c56bcebbbe98c6bb7478a2\": container with ID starting with 4ad2bf2790cbf968d8fd8a69b5861c34e0666a2749c56bcebbbe98c6bb7478a2 not found: ID does not exist"
Sep 16 13:09:52.663884 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.663863    2577 scope.go:117] "RemoveContainer" containerID="4597b6bb5db08ddf331310f64c8e1f1529b730ffd6466a285d45ef1b97e58640"
Sep 16 13:09:52.664258 ip-10-0-133-190 kubenswrapper[2577]: E0916 13:09:52.664233    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4597b6bb5db08ddf331310f64c8e1f1529b730ffd6466a285d45ef1b97e58640\": container with ID starting with 4597b6bb5db08ddf331310f64c8e1f1529b730ffd6466a285d45ef1b97e58640 not found: ID does not exist" containerID="4597b6bb5db08ddf331310f64c8e1f1529b730ffd6466a285d45ef1b97e58640"
Sep 16 13:09:52.664374 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.664264    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4597b6bb5db08ddf331310f64c8e1f1529b730ffd6466a285d45ef1b97e58640"} err="failed to get container status \"4597b6bb5db08ddf331310f64c8e1f1529b730ffd6466a285d45ef1b97e58640\": rpc error: code = NotFound desc = could not find container \"4597b6bb5db08ddf331310f64c8e1f1529b730ffd6466a285d45ef1b97e58640\": container with ID starting with 4597b6bb5db08ddf331310f64c8e1f1529b730ffd6466a285d45ef1b97e58640 not found: ID does not exist"
Sep 16 13:09:52.666900 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.666879    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"]
Sep 16 13:09:52.670862 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.670842    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e5926f-openshift-default-db84df5cd-l9lmn"]
Sep 16 13:09:52.699266 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.699241    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05103a7a-76fc-4417-a62a-2d1439edc444-istio-podinfo\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:52.699266 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.699263    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05103a7a-76fc-4417-a62a-2d1439edc444-istio-token\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:52.699395 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.699272    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7ffw\" (UniqueName: \"kubernetes.io/projected/05103a7a-76fc-4417-a62a-2d1439edc444-kube-api-access-r7ffw\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:52.699395 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:52.699281    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05103a7a-76fc-4417-a62a-2d1439edc444-istio-envoy\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:09:54.054930 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:09:54.054898    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05103a7a-76fc-4417-a62a-2d1439edc444" path="/var/lib/kubelet/pods/05103a7a-76fc-4417-a62a-2d1439edc444/volumes"
Sep 16 13:10:11.678603 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.678567    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-e344f9/sub-test-model-e344f9-kserve-75f5798df7-5qtfk"]
Sep 16 13:10:11.679295 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.679230    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08e00b87-97af-456a-af30-10b145da3ea6" containerName="main"
Sep 16 13:10:11.679295 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.679256    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="08e00b87-97af-456a-af30-10b145da3ea6" containerName="main"
Sep 16 13:10:11.679295 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.679273    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7735aca7-7d6b-464d-872a-00a4709feaec" containerName="main"
Sep 16 13:10:11.679295 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.679292    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7735aca7-7d6b-464d-872a-00a4709feaec" containerName="main"
Sep 16 13:10:11.679528 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.679307    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99b70637-81ce-4503-bc3d-de5dc69e7b97" containerName="istio-proxy"
Sep 16 13:10:11.679528 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.679316    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="99b70637-81ce-4503-bc3d-de5dc69e7b97" containerName="istio-proxy"
Sep 16 13:10:11.679528 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.679331    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01d14db3-8e1d-4462-a9a6-f92962543f90" containerName="payload-pre-processing"
Sep 16 13:10:11.679528 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.679340    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="01d14db3-8e1d-4462-a9a6-f92962543f90" containerName="payload-pre-processing"
Sep 16 13:10:11.679528 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.679357    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01a3c352-0789-42df-8d81-3242438a84c2" containerName="payload-processing"
Sep 16 13:10:11.679528 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.679365    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="01a3c352-0789-42df-8d81-3242438a84c2" containerName="payload-processing"
Sep 16 13:10:11.679528 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.679379    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99b70637-81ce-4503-bc3d-de5dc69e7b97" containerName="istio-proxy"
Sep 16 13:10:11.679528 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.679387    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="99b70637-81ce-4503-bc3d-de5dc69e7b97" containerName="istio-proxy"
Sep 16 13:10:11.679528 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.679397    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4559413e-b78b-4499-b85a-81ad046f306f" containerName="revoke-keys"
Sep 16 13:10:11.679528 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.679405    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4559413e-b78b-4499-b85a-81ad046f306f" containerName="revoke-keys"
Sep 16 13:10:11.679528 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.679415    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05103a7a-76fc-4417-a62a-2d1439edc444" containerName="istio-proxy"
Sep 16 13:10:11.679528 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.679423    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="05103a7a-76fc-4417-a62a-2d1439edc444" containerName="istio-proxy"
Sep 16 13:10:11.679528 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.679435    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ddfa698-b491-497a-be0c-602cb4c1f8ed" containerName="main"
Sep 16 13:10:11.679528 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.679443    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ddfa698-b491-497a-be0c-602cb4c1f8ed" containerName="main"
Sep 16 13:10:11.679528 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.679455    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05103a7a-76fc-4417-a62a-2d1439edc444" containerName="istio-proxy"
Sep 16 13:10:11.679528 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.679479    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="05103a7a-76fc-4417-a62a-2d1439edc444" containerName="istio-proxy"
Sep 16 13:10:11.680313 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.679608    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="05103a7a-76fc-4417-a62a-2d1439edc444" containerName="istio-proxy"
Sep 16 13:10:11.680313 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.679622    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ddfa698-b491-497a-be0c-602cb4c1f8ed" containerName="main"
Sep 16 13:10:11.680313 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.679635    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="99b70637-81ce-4503-bc3d-de5dc69e7b97" containerName="istio-proxy"
Sep 16 13:10:11.680313 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.679643    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="08e00b87-97af-456a-af30-10b145da3ea6" containerName="main"
Sep 16 13:10:11.680313 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.679650    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7735aca7-7d6b-464d-872a-00a4709feaec" containerName="main"
Sep 16 13:10:11.680313 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.679660    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4559413e-b78b-4499-b85a-81ad046f306f" containerName="revoke-keys"
Sep 16 13:10:11.680313 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.679669    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="01d14db3-8e1d-4462-a9a6-f92962543f90" containerName="payload-pre-processing"
Sep 16 13:10:11.680313 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.679676    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="01a3c352-0789-42df-8d81-3242438a84c2" containerName="payload-processing"
Sep 16 13:10:11.682915 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.682894    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e344f9/sub-test-model-e344f9-kserve-75f5798df7-5qtfk"
Sep 16 13:10:11.684355 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.684336    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2240641-e81a-4a91-808d-7da4e3c364d9-dshm\") pod \"sub-test-model-e344f9-kserve-75f5798df7-5qtfk\" (UID: \"b2240641-e81a-4a91-808d-7da4e3c364d9\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/sub-test-model-e344f9-kserve-75f5798df7-5qtfk"
Sep 16 13:10:11.684447 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.684368    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2240641-e81a-4a91-808d-7da4e3c364d9-model-cache\") pod \"sub-test-model-e344f9-kserve-75f5798df7-5qtfk\" (UID: \"b2240641-e81a-4a91-808d-7da4e3c364d9\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/sub-test-model-e344f9-kserve-75f5798df7-5qtfk"
Sep 16 13:10:11.684447 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.684387    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2240641-e81a-4a91-808d-7da4e3c364d9-home\") pod \"sub-test-model-e344f9-kserve-75f5798df7-5qtfk\" (UID: \"b2240641-e81a-4a91-808d-7da4e3c364d9\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/sub-test-model-e344f9-kserve-75f5798df7-5qtfk"
Sep 16 13:10:11.684527 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.684481    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2240641-e81a-4a91-808d-7da4e3c364d9-tmp-dir\") pod \"sub-test-model-e344f9-kserve-75f5798df7-5qtfk\" (UID: \"b2240641-e81a-4a91-808d-7da4e3c364d9\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/sub-test-model-e344f9-kserve-75f5798df7-5qtfk"
Sep 16 13:10:11.684527 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.684511    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jb4l\" (UniqueName: \"kubernetes.io/projected/b2240641-e81a-4a91-808d-7da4e3c364d9-kube-api-access-2jb4l\") pod \"sub-test-model-e344f9-kserve-75f5798df7-5qtfk\" (UID: \"b2240641-e81a-4a91-808d-7da4e3c364d9\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/sub-test-model-e344f9-kserve-75f5798df7-5qtfk"
Sep 16 13:10:11.684593 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.684548    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2240641-e81a-4a91-808d-7da4e3c364d9-tls-certs\") pod \"sub-test-model-e344f9-kserve-75f5798df7-5qtfk\" (UID: \"b2240641-e81a-4a91-808d-7da4e3c364d9\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/sub-test-model-e344f9-kserve-75f5798df7-5qtfk"
Sep 16 13:10:11.688265 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.688246    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e344f9\"/\"kube-root-ca.crt\""
Sep 16 13:10:11.688570 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.688551    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e344f9\"/\"default-dockercfg-gwp2c\""
Sep 16 13:10:11.688678 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.688598    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e344f9\"/\"openshift-service-ca.crt\""
Sep 16 13:10:11.688678 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.688606    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e344f9\"/\"sub-test-model-e344f9-kserve-self-signed-certs\""
Sep 16 13:10:11.695452 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.695431    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e344f9/sub-test-model-e344f9-kserve-75f5798df7-5qtfk"]
Sep 16 13:10:11.785407 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.785378    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2240641-e81a-4a91-808d-7da4e3c364d9-tls-certs\") pod \"sub-test-model-e344f9-kserve-75f5798df7-5qtfk\" (UID: \"b2240641-e81a-4a91-808d-7da4e3c364d9\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/sub-test-model-e344f9-kserve-75f5798df7-5qtfk"
Sep 16 13:10:11.785619 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.785466    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2240641-e81a-4a91-808d-7da4e3c364d9-dshm\") pod \"sub-test-model-e344f9-kserve-75f5798df7-5qtfk\" (UID: \"b2240641-e81a-4a91-808d-7da4e3c364d9\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/sub-test-model-e344f9-kserve-75f5798df7-5qtfk"
Sep 16 13:10:11.785619 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.785498    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2240641-e81a-4a91-808d-7da4e3c364d9-model-cache\") pod \"sub-test-model-e344f9-kserve-75f5798df7-5qtfk\" (UID: \"b2240641-e81a-4a91-808d-7da4e3c364d9\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/sub-test-model-e344f9-kserve-75f5798df7-5qtfk"
Sep 16 13:10:11.785619 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.785524    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2240641-e81a-4a91-808d-7da4e3c364d9-home\") pod \"sub-test-model-e344f9-kserve-75f5798df7-5qtfk\" (UID: \"b2240641-e81a-4a91-808d-7da4e3c364d9\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/sub-test-model-e344f9-kserve-75f5798df7-5qtfk"
Sep 16 13:10:11.785817 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.785671    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2240641-e81a-4a91-808d-7da4e3c364d9-tmp-dir\") pod \"sub-test-model-e344f9-kserve-75f5798df7-5qtfk\" (UID: \"b2240641-e81a-4a91-808d-7da4e3c364d9\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/sub-test-model-e344f9-kserve-75f5798df7-5qtfk"
Sep 16 13:10:11.785817 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.785709    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jb4l\" (UniqueName: \"kubernetes.io/projected/b2240641-e81a-4a91-808d-7da4e3c364d9-kube-api-access-2jb4l\") pod \"sub-test-model-e344f9-kserve-75f5798df7-5qtfk\" (UID: \"b2240641-e81a-4a91-808d-7da4e3c364d9\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/sub-test-model-e344f9-kserve-75f5798df7-5qtfk"
Sep 16 13:10:11.785962 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.785939    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2240641-e81a-4a91-808d-7da4e3c364d9-home\") pod \"sub-test-model-e344f9-kserve-75f5798df7-5qtfk\" (UID: \"b2240641-e81a-4a91-808d-7da4e3c364d9\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/sub-test-model-e344f9-kserve-75f5798df7-5qtfk"
Sep 16 13:10:11.786017 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.785978    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2240641-e81a-4a91-808d-7da4e3c364d9-model-cache\") pod \"sub-test-model-e344f9-kserve-75f5798df7-5qtfk\" (UID: \"b2240641-e81a-4a91-808d-7da4e3c364d9\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/sub-test-model-e344f9-kserve-75f5798df7-5qtfk"
Sep 16 13:10:11.786087 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.786072    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2240641-e81a-4a91-808d-7da4e3c364d9-tmp-dir\") pod \"sub-test-model-e344f9-kserve-75f5798df7-5qtfk\" (UID: \"b2240641-e81a-4a91-808d-7da4e3c364d9\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/sub-test-model-e344f9-kserve-75f5798df7-5qtfk"
Sep 16 13:10:11.788568 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.788540    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2240641-e81a-4a91-808d-7da4e3c364d9-dshm\") pod \"sub-test-model-e344f9-kserve-75f5798df7-5qtfk\" (UID: \"b2240641-e81a-4a91-808d-7da4e3c364d9\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/sub-test-model-e344f9-kserve-75f5798df7-5qtfk"
Sep 16 13:10:11.788773 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.788755    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2240641-e81a-4a91-808d-7da4e3c364d9-tls-certs\") pod \"sub-test-model-e344f9-kserve-75f5798df7-5qtfk\" (UID: \"b2240641-e81a-4a91-808d-7da4e3c364d9\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/sub-test-model-e344f9-kserve-75f5798df7-5qtfk"
Sep 16 13:10:11.796428 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.796402    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jb4l\" (UniqueName: \"kubernetes.io/projected/b2240641-e81a-4a91-808d-7da4e3c364d9-kube-api-access-2jb4l\") pod \"sub-test-model-e344f9-kserve-75f5798df7-5qtfk\" (UID: \"b2240641-e81a-4a91-808d-7da4e3c364d9\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/sub-test-model-e344f9-kserve-75f5798df7-5qtfk"
Sep 16 13:10:11.992939 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:11.992844    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e344f9/sub-test-model-e344f9-kserve-75f5798df7-5qtfk"
Sep 16 13:10:12.277525 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:12.277469    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e344f9/sub-test-model-e344f9-kserve-75f5798df7-5qtfk"]
Sep 16 13:10:12.280195 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:10:12.280167    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2240641_e81a_4a91_808d_7da4e3c364d9.slice/crio-460ae9cc5ff8dd826631d562368e11e62577444804f75831501698313c45c7c9 WatchSource:0}: Error finding container 460ae9cc5ff8dd826631d562368e11e62577444804f75831501698313c45c7c9: Status 404 returned error can't find the container with id 460ae9cc5ff8dd826631d562368e11e62577444804f75831501698313c45c7c9
Sep 16 13:10:12.763894 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:12.763857    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e344f9/sub-test-model-e344f9-kserve-75f5798df7-5qtfk" event={"ID":"b2240641-e81a-4a91-808d-7da4e3c364d9","Type":"ContainerStarted","Data":"7c93b899f56fa2d441a15f988dc72627108c79f22f25ec2775f094b600e48fb5"}
Sep 16 13:10:12.764328 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:12.763906    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e344f9/sub-test-model-e344f9-kserve-75f5798df7-5qtfk" event={"ID":"b2240641-e81a-4a91-808d-7da4e3c364d9","Type":"ContainerStarted","Data":"460ae9cc5ff8dd826631d562368e11e62577444804f75831501698313c45c7c9"}
Sep 16 13:10:12.798652 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:12.798597    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-e344f9/sub-test-model-e344f9-kserve-75f5798df7-5qtfk" podStartSLOduration=1.7985811520000001 podStartE2EDuration="1.798581152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:10:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:10:12.797173658 +0000 UTC m=+1919.244406300" watchObservedRunningTime="2026-09-16 13:10:12.798581152 +0000 UTC m=+1919.245813795"
Sep 16 13:10:15.029083 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:15.029017    2577 scope.go:117] "RemoveContainer" containerID="ef1e64ac850f62711beea6b04a8450f4580e19e9e3e04900e842cf40a62e1cd6"
Sep 16 13:10:15.038092 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:15.038069    2577 scope.go:117] "RemoveContainer" containerID="14bffd5b0307c685226e090273dc887d6e0c0815bb983dcef7d986efc4b44f41"
Sep 16 13:10:21.993197 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:21.993160    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-e344f9/sub-test-model-e344f9-kserve-75f5798df7-5qtfk"
Sep 16 13:10:21.993639 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:21.993214    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-e344f9/sub-test-model-e344f9-kserve-75f5798df7-5qtfk"
Sep 16 13:10:22.006089 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:22.006065    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-e344f9/sub-test-model-e344f9-kserve-75f5798df7-5qtfk"
Sep 16 13:10:22.834269 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:22.834238    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-e344f9/sub-test-model-e344f9-kserve-75f5798df7-5qtfk"
Sep 16 13:10:48.331816 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:10:48.331784    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e344f9/sub-test-model-e344f9-kserve-75f5798df7-5qtfk"]
Sep 16 13:11:03.351910 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:03.351848    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-e344f9/sub-test-model-e344f9-kserve-75f5798df7-5qtfk" podUID="b2240641-e81a-4a91-808d-7da4e3c364d9" containerName="main" containerID="cri-o://7c93b899f56fa2d441a15f988dc72627108c79f22f25ec2775f094b600e48fb5" gracePeriod=45
Sep 16 13:11:03.603752 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:03.603688    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e344f9/sub-test-model-e344f9-kserve-75f5798df7-5qtfk"
Sep 16 13:11:03.679884 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:03.679852    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2240641-e81a-4a91-808d-7da4e3c364d9-dshm\") pod \"b2240641-e81a-4a91-808d-7da4e3c364d9\" (UID: \"b2240641-e81a-4a91-808d-7da4e3c364d9\") "
Sep 16 13:11:03.680116 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:03.679900    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jb4l\" (UniqueName: \"kubernetes.io/projected/b2240641-e81a-4a91-808d-7da4e3c364d9-kube-api-access-2jb4l\") pod \"b2240641-e81a-4a91-808d-7da4e3c364d9\" (UID: \"b2240641-e81a-4a91-808d-7da4e3c364d9\") "
Sep 16 13:11:03.680116 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:03.679935    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2240641-e81a-4a91-808d-7da4e3c364d9-tmp-dir\") pod \"b2240641-e81a-4a91-808d-7da4e3c364d9\" (UID: \"b2240641-e81a-4a91-808d-7da4e3c364d9\") "
Sep 16 13:11:03.680116 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:03.680019    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2240641-e81a-4a91-808d-7da4e3c364d9-tls-certs\") pod \"b2240641-e81a-4a91-808d-7da4e3c364d9\" (UID: \"b2240641-e81a-4a91-808d-7da4e3c364d9\") "
Sep 16 13:11:03.680116 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:03.680075    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2240641-e81a-4a91-808d-7da4e3c364d9-home\") pod \"b2240641-e81a-4a91-808d-7da4e3c364d9\" (UID: \"b2240641-e81a-4a91-808d-7da4e3c364d9\") "
Sep 16 13:11:03.680116 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:03.680105    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2240641-e81a-4a91-808d-7da4e3c364d9-model-cache\") pod \"b2240641-e81a-4a91-808d-7da4e3c364d9\" (UID: \"b2240641-e81a-4a91-808d-7da4e3c364d9\") "
Sep 16 13:11:03.680391 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:03.680332    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2240641-e81a-4a91-808d-7da4e3c364d9-home" (OuterVolumeSpecName: "home") pod "b2240641-e81a-4a91-808d-7da4e3c364d9" (UID: "b2240641-e81a-4a91-808d-7da4e3c364d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:11:03.680391 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:03.680347    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2240641-e81a-4a91-808d-7da4e3c364d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b2240641-e81a-4a91-808d-7da4e3c364d9" (UID: "b2240641-e81a-4a91-808d-7da4e3c364d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:11:03.680391 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:03.680362    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2240641-e81a-4a91-808d-7da4e3c364d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "b2240641-e81a-4a91-808d-7da4e3c364d9" (UID: "b2240641-e81a-4a91-808d-7da4e3c364d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:11:03.680602 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:03.680464    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2240641-e81a-4a91-808d-7da4e3c364d9-model-cache\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:11:03.680602 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:03.680477    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2240641-e81a-4a91-808d-7da4e3c364d9-tmp-dir\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:11:03.680602 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:03.680485    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2240641-e81a-4a91-808d-7da4e3c364d9-home\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:11:03.682361 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:03.682335    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2240641-e81a-4a91-808d-7da4e3c364d9-dshm" (OuterVolumeSpecName: "dshm") pod "b2240641-e81a-4a91-808d-7da4e3c364d9" (UID: "b2240641-e81a-4a91-808d-7da4e3c364d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:11:03.682361 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:03.682356    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2240641-e81a-4a91-808d-7da4e3c364d9-kube-api-access-2jb4l" (OuterVolumeSpecName: "kube-api-access-2jb4l") pod "b2240641-e81a-4a91-808d-7da4e3c364d9" (UID: "b2240641-e81a-4a91-808d-7da4e3c364d9"). InnerVolumeSpecName "kube-api-access-2jb4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:11:03.682508 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:03.682366    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2240641-e81a-4a91-808d-7da4e3c364d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b2240641-e81a-4a91-808d-7da4e3c364d9" (UID: "b2240641-e81a-4a91-808d-7da4e3c364d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:11:03.781852 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:03.781820    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2240641-e81a-4a91-808d-7da4e3c364d9-dshm\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:11:03.781852 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:03.781847    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jb4l\" (UniqueName: \"kubernetes.io/projected/b2240641-e81a-4a91-808d-7da4e3c364d9-kube-api-access-2jb4l\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:11:03.781852 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:03.781858    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2240641-e81a-4a91-808d-7da4e3c364d9-tls-certs\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:11:04.049619 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:04.049586    2577 generic.go:356] "Generic (PLEG): container finished" podID="b2240641-e81a-4a91-808d-7da4e3c364d9" containerID="7c93b899f56fa2d441a15f988dc72627108c79f22f25ec2775f094b600e48fb5" exitCode=0
Sep 16 13:11:04.051776 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:04.051755    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e344f9/sub-test-model-e344f9-kserve-75f5798df7-5qtfk"
Sep 16 13:11:04.055949 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:04.055920    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e344f9/sub-test-model-e344f9-kserve-75f5798df7-5qtfk" event={"ID":"b2240641-e81a-4a91-808d-7da4e3c364d9","Type":"ContainerDied","Data":"7c93b899f56fa2d441a15f988dc72627108c79f22f25ec2775f094b600e48fb5"}
Sep 16 13:11:04.056095 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:04.055960    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e344f9/sub-test-model-e344f9-kserve-75f5798df7-5qtfk" event={"ID":"b2240641-e81a-4a91-808d-7da4e3c364d9","Type":"ContainerDied","Data":"460ae9cc5ff8dd826631d562368e11e62577444804f75831501698313c45c7c9"}
Sep 16 13:11:04.056095 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:04.055985    2577 scope.go:117] "RemoveContainer" containerID="7c93b899f56fa2d441a15f988dc72627108c79f22f25ec2775f094b600e48fb5"
Sep 16 13:11:04.066588 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:04.066573    2577 scope.go:117] "RemoveContainer" containerID="7c93b899f56fa2d441a15f988dc72627108c79f22f25ec2775f094b600e48fb5"
Sep 16 13:11:04.066856 ip-10-0-133-190 kubenswrapper[2577]: E0916 13:11:04.066838    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c93b899f56fa2d441a15f988dc72627108c79f22f25ec2775f094b600e48fb5\": container with ID starting with 7c93b899f56fa2d441a15f988dc72627108c79f22f25ec2775f094b600e48fb5 not found: ID does not exist" containerID="7c93b899f56fa2d441a15f988dc72627108c79f22f25ec2775f094b600e48fb5"
Sep 16 13:11:04.066898 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:04.066865    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c93b899f56fa2d441a15f988dc72627108c79f22f25ec2775f094b600e48fb5"} err="failed to get container status \"7c93b899f56fa2d441a15f988dc72627108c79f22f25ec2775f094b600e48fb5\": rpc error: code = NotFound desc = could not find container \"7c93b899f56fa2d441a15f988dc72627108c79f22f25ec2775f094b600e48fb5\": container with ID starting with 7c93b899f56fa2d441a15f988dc72627108c79f22f25ec2775f094b600e48fb5 not found: ID does not exist"
Sep 16 13:11:04.077733 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:04.077710    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e344f9/sub-test-model-e344f9-kserve-75f5798df7-5qtfk"]
Sep 16 13:11:04.079625 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:04.079606    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e344f9/sub-test-model-e344f9-kserve-75f5798df7-5qtfk"]
Sep 16 13:11:06.055603 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:06.055524    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2240641-e81a-4a91-808d-7da4e3c364d9" path="/var/lib/kubelet/pods/b2240641-e81a-4a91-808d-7da4e3c364d9/volumes"
Sep 16 13:11:15.091293 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:15.091259    2577 scope.go:117] "RemoveContainer" containerID="aa147a812b8491206e69829eedfeae2ff45c5bc43ca651a446bfc8d58f1349ad"
Sep 16 13:11:26.955402 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:26.955367    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-e344f9/rate-test-model-e344f9-kserve-67455fcdcf-7cpq2"]
Sep 16 13:11:26.955862 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:26.955782    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2240641-e81a-4a91-808d-7da4e3c364d9" containerName="main"
Sep 16 13:11:26.955862 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:26.955793    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2240641-e81a-4a91-808d-7da4e3c364d9" containerName="main"
Sep 16 13:11:26.955862 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:26.955858    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="05103a7a-76fc-4417-a62a-2d1439edc444" containerName="istio-proxy"
Sep 16 13:11:26.955966 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:26.955868    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2240641-e81a-4a91-808d-7da4e3c364d9" containerName="main"
Sep 16 13:11:26.955966 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:26.955877    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="99b70637-81ce-4503-bc3d-de5dc69e7b97" containerName="istio-proxy"
Sep 16 13:11:26.959148 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:26.959125    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e344f9/rate-test-model-e344f9-kserve-67455fcdcf-7cpq2"
Sep 16 13:11:26.963091 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:26.963064    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e344f9\"/\"kube-root-ca.crt\""
Sep 16 13:11:26.963216 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:26.963120    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e344f9\"/\"openshift-service-ca.crt\""
Sep 16 13:11:26.963280 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:26.963122    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e344f9\"/\"rate-test-model-e344f9-kserve-self-signed-certs\""
Sep 16 13:11:26.963280 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:26.963064    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e344f9\"/\"default-dockercfg-gwp2c\""
Sep 16 13:11:26.967816 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:26.967787    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e344f9/rate-test-model-e344f9-kserve-67455fcdcf-7cpq2"]
Sep 16 13:11:27.091052 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:27.090996    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c48ef941-6fe7-45a5-8ac7-54852d99643f-dshm\") pod \"rate-test-model-e344f9-kserve-67455fcdcf-7cpq2\" (UID: \"c48ef941-6fe7-45a5-8ac7-54852d99643f\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/rate-test-model-e344f9-kserve-67455fcdcf-7cpq2"
Sep 16 13:11:27.091052 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:27.091047    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c48ef941-6fe7-45a5-8ac7-54852d99643f-tls-certs\") pod \"rate-test-model-e344f9-kserve-67455fcdcf-7cpq2\" (UID: \"c48ef941-6fe7-45a5-8ac7-54852d99643f\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/rate-test-model-e344f9-kserve-67455fcdcf-7cpq2"
Sep 16 13:11:27.091323 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:27.091068    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c48ef941-6fe7-45a5-8ac7-54852d99643f-home\") pod \"rate-test-model-e344f9-kserve-67455fcdcf-7cpq2\" (UID: \"c48ef941-6fe7-45a5-8ac7-54852d99643f\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/rate-test-model-e344f9-kserve-67455fcdcf-7cpq2"
Sep 16 13:11:27.091323 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:27.091207    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dnsh\" (UniqueName: \"kubernetes.io/projected/c48ef941-6fe7-45a5-8ac7-54852d99643f-kube-api-access-2dnsh\") pod \"rate-test-model-e344f9-kserve-67455fcdcf-7cpq2\" (UID: \"c48ef941-6fe7-45a5-8ac7-54852d99643f\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/rate-test-model-e344f9-kserve-67455fcdcf-7cpq2"
Sep 16 13:11:27.091323 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:27.091264    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c48ef941-6fe7-45a5-8ac7-54852d99643f-model-cache\") pod \"rate-test-model-e344f9-kserve-67455fcdcf-7cpq2\" (UID: \"c48ef941-6fe7-45a5-8ac7-54852d99643f\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/rate-test-model-e344f9-kserve-67455fcdcf-7cpq2"
Sep 16 13:11:27.091323 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:27.091313    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c48ef941-6fe7-45a5-8ac7-54852d99643f-tmp-dir\") pod \"rate-test-model-e344f9-kserve-67455fcdcf-7cpq2\" (UID: \"c48ef941-6fe7-45a5-8ac7-54852d99643f\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/rate-test-model-e344f9-kserve-67455fcdcf-7cpq2"
Sep 16 13:11:27.192822 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:27.192783    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c48ef941-6fe7-45a5-8ac7-54852d99643f-tmp-dir\") pod \"rate-test-model-e344f9-kserve-67455fcdcf-7cpq2\" (UID: \"c48ef941-6fe7-45a5-8ac7-54852d99643f\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/rate-test-model-e344f9-kserve-67455fcdcf-7cpq2"
Sep 16 13:11:27.193041 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:27.192859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c48ef941-6fe7-45a5-8ac7-54852d99643f-dshm\") pod \"rate-test-model-e344f9-kserve-67455fcdcf-7cpq2\" (UID: \"c48ef941-6fe7-45a5-8ac7-54852d99643f\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/rate-test-model-e344f9-kserve-67455fcdcf-7cpq2"
Sep 16 13:11:27.193041 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:27.192880    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c48ef941-6fe7-45a5-8ac7-54852d99643f-tls-certs\") pod \"rate-test-model-e344f9-kserve-67455fcdcf-7cpq2\" (UID: \"c48ef941-6fe7-45a5-8ac7-54852d99643f\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/rate-test-model-e344f9-kserve-67455fcdcf-7cpq2"
Sep 16 13:11:27.193041 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:27.192895    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c48ef941-6fe7-45a5-8ac7-54852d99643f-home\") pod \"rate-test-model-e344f9-kserve-67455fcdcf-7cpq2\" (UID: \"c48ef941-6fe7-45a5-8ac7-54852d99643f\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/rate-test-model-e344f9-kserve-67455fcdcf-7cpq2"
Sep 16 13:11:27.193041 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:27.192935    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dnsh\" (UniqueName: \"kubernetes.io/projected/c48ef941-6fe7-45a5-8ac7-54852d99643f-kube-api-access-2dnsh\") pod \"rate-test-model-e344f9-kserve-67455fcdcf-7cpq2\" (UID: \"c48ef941-6fe7-45a5-8ac7-54852d99643f\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/rate-test-model-e344f9-kserve-67455fcdcf-7cpq2"
Sep 16 13:11:27.193041 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:27.192960    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c48ef941-6fe7-45a5-8ac7-54852d99643f-model-cache\") pod \"rate-test-model-e344f9-kserve-67455fcdcf-7cpq2\" (UID: \"c48ef941-6fe7-45a5-8ac7-54852d99643f\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/rate-test-model-e344f9-kserve-67455fcdcf-7cpq2"
Sep 16 13:11:27.193345 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:27.193272    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c48ef941-6fe7-45a5-8ac7-54852d99643f-tmp-dir\") pod \"rate-test-model-e344f9-kserve-67455fcdcf-7cpq2\" (UID: \"c48ef941-6fe7-45a5-8ac7-54852d99643f\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/rate-test-model-e344f9-kserve-67455fcdcf-7cpq2"
Sep 16 13:11:27.193388 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:27.193344    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c48ef941-6fe7-45a5-8ac7-54852d99643f-home\") pod \"rate-test-model-e344f9-kserve-67455fcdcf-7cpq2\" (UID: \"c48ef941-6fe7-45a5-8ac7-54852d99643f\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/rate-test-model-e344f9-kserve-67455fcdcf-7cpq2"
Sep 16 13:11:27.193424 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:27.193411    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c48ef941-6fe7-45a5-8ac7-54852d99643f-model-cache\") pod \"rate-test-model-e344f9-kserve-67455fcdcf-7cpq2\" (UID: \"c48ef941-6fe7-45a5-8ac7-54852d99643f\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/rate-test-model-e344f9-kserve-67455fcdcf-7cpq2"
Sep 16 13:11:27.195451 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:27.195427    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c48ef941-6fe7-45a5-8ac7-54852d99643f-dshm\") pod \"rate-test-model-e344f9-kserve-67455fcdcf-7cpq2\" (UID: \"c48ef941-6fe7-45a5-8ac7-54852d99643f\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/rate-test-model-e344f9-kserve-67455fcdcf-7cpq2"
Sep 16 13:11:27.195611 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:27.195593    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c48ef941-6fe7-45a5-8ac7-54852d99643f-tls-certs\") pod \"rate-test-model-e344f9-kserve-67455fcdcf-7cpq2\" (UID: \"c48ef941-6fe7-45a5-8ac7-54852d99643f\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/rate-test-model-e344f9-kserve-67455fcdcf-7cpq2"
Sep 16 13:11:27.200604 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:27.200582    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dnsh\" (UniqueName: \"kubernetes.io/projected/c48ef941-6fe7-45a5-8ac7-54852d99643f-kube-api-access-2dnsh\") pod \"rate-test-model-e344f9-kserve-67455fcdcf-7cpq2\" (UID: \"c48ef941-6fe7-45a5-8ac7-54852d99643f\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/rate-test-model-e344f9-kserve-67455fcdcf-7cpq2"
Sep 16 13:11:27.272103 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:27.272068    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e344f9/rate-test-model-e344f9-kserve-67455fcdcf-7cpq2"
Sep 16 13:11:27.553090 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:27.553059    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e344f9/rate-test-model-e344f9-kserve-67455fcdcf-7cpq2"]
Sep 16 13:11:27.555018 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:11:27.554989    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc48ef941_6fe7_45a5_8ac7_54852d99643f.slice/crio-e6d2b5369156924952ed83179260187bc003326c5feb096de788cada956877e2 WatchSource:0}: Error finding container e6d2b5369156924952ed83179260187bc003326c5feb096de788cada956877e2: Status 404 returned error can't find the container with id e6d2b5369156924952ed83179260187bc003326c5feb096de788cada956877e2
Sep 16 13:11:28.180226 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:28.180189    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e344f9/rate-test-model-e344f9-kserve-67455fcdcf-7cpq2" event={"ID":"c48ef941-6fe7-45a5-8ac7-54852d99643f","Type":"ContainerStarted","Data":"53197c0c10f377d9b1574690d4615d02923c7756362c45f1634ce12cf7c40c7d"}
Sep 16 13:11:28.180628 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:28.180234    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e344f9/rate-test-model-e344f9-kserve-67455fcdcf-7cpq2" event={"ID":"c48ef941-6fe7-45a5-8ac7-54852d99643f","Type":"ContainerStarted","Data":"e6d2b5369156924952ed83179260187bc003326c5feb096de788cada956877e2"}
Sep 16 13:11:28.198049 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:28.197970    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-e344f9/rate-test-model-e344f9-kserve-67455fcdcf-7cpq2" podStartSLOduration=2.197948437 podStartE2EDuration="2.197948437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:11:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:11:28.196149217 +0000 UTC m=+1994.643381861" watchObservedRunningTime="2026-09-16 13:11:28.197948437 +0000 UTC m=+1994.645181079"
Sep 16 13:11:37.272496 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:37.272457    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-e344f9/rate-test-model-e344f9-kserve-67455fcdcf-7cpq2"
Sep 16 13:11:37.272496 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:37.272502    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-e344f9/rate-test-model-e344f9-kserve-67455fcdcf-7cpq2"
Sep 16 13:11:37.285051 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:37.285005    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-e344f9/rate-test-model-e344f9-kserve-67455fcdcf-7cpq2"
Sep 16 13:11:38.245804 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:11:38.245773    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-e344f9/rate-test-model-e344f9-kserve-67455fcdcf-7cpq2"
Sep 16 13:12:18.838388 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:18.838353    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e344f9/rate-test-model-e344f9-kserve-67455fcdcf-7cpq2"]
Sep 16 13:12:24.760810 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:24.760771    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a17de2-669fc94f85-t5cql"]
Sep 16 13:12:24.764471 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:24.764453    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a17de2-669fc94f85-t5cql"
Sep 16 13:12:24.767318 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:24.767295    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a17de2\""
Sep 16 13:12:24.767425 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:24.767339    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a17de2-dockercfg-vhxzt\""
Sep 16 13:12:24.775902 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:24.775882    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a17de2-669fc94f85-t5cql"]
Sep 16 13:12:24.808341 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:24.808314    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a17de2-54494c4c98-q8gtg"]
Sep 16 13:12:24.811872 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:24.811849    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a17de2-54494c4c98-q8gtg"
Sep 16 13:12:24.820636 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:24.820609    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a17de2-54494c4c98-q8gtg"]
Sep 16 13:12:24.865689 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:24.865657    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c47bfb-9cfc-4105-8f1b-9b2a505e68ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a17de2-54494c4c98-q8gtg\" (UID: \"20c47bfb-9cfc-4105-8f1b-9b2a505e68ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a17de2-54494c4c98-q8gtg"
Sep 16 13:12:24.865689 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:24.865692    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf4ed91e-8b4d-41a4-be55-9fa3da6a9302-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a17de2-669fc94f85-t5cql\" (UID: \"bf4ed91e-8b4d-41a4-be55-9fa3da6a9302\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a17de2-669fc94f85-t5cql"
Sep 16 13:12:24.865911 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:24.865723    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rtvp\" (UniqueName: \"kubernetes.io/projected/20c47bfb-9cfc-4105-8f1b-9b2a505e68ec-kube-api-access-6rtvp\") pod \"payload-pre-processing-e2e-ipp-a-a17de2-54494c4c98-q8gtg\" (UID: \"20c47bfb-9cfc-4105-8f1b-9b2a505e68ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a17de2-54494c4c98-q8gtg"
Sep 16 13:12:24.865911 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:24.865843    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxw79\" (UniqueName: \"kubernetes.io/projected/bf4ed91e-8b4d-41a4-be55-9fa3da6a9302-kube-api-access-cxw79\") pod \"payload-processing-e2e-ipp-a-a17de2-669fc94f85-t5cql\" (UID: \"bf4ed91e-8b4d-41a4-be55-9fa3da6a9302\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a17de2-669fc94f85-t5cql"
Sep 16 13:12:24.967044 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:24.966991    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c47bfb-9cfc-4105-8f1b-9b2a505e68ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a17de2-54494c4c98-q8gtg\" (UID: \"20c47bfb-9cfc-4105-8f1b-9b2a505e68ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a17de2-54494c4c98-q8gtg"
Sep 16 13:12:24.967232 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:24.967055    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf4ed91e-8b4d-41a4-be55-9fa3da6a9302-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a17de2-669fc94f85-t5cql\" (UID: \"bf4ed91e-8b4d-41a4-be55-9fa3da6a9302\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a17de2-669fc94f85-t5cql"
Sep 16 13:12:24.967232 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:24.967088    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rtvp\" (UniqueName: \"kubernetes.io/projected/20c47bfb-9cfc-4105-8f1b-9b2a505e68ec-kube-api-access-6rtvp\") pod \"payload-pre-processing-e2e-ipp-a-a17de2-54494c4c98-q8gtg\" (UID: \"20c47bfb-9cfc-4105-8f1b-9b2a505e68ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a17de2-54494c4c98-q8gtg"
Sep 16 13:12:24.967232 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:24.967175    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxw79\" (UniqueName: \"kubernetes.io/projected/bf4ed91e-8b4d-41a4-be55-9fa3da6a9302-kube-api-access-cxw79\") pod \"payload-processing-e2e-ipp-a-a17de2-669fc94f85-t5cql\" (UID: \"bf4ed91e-8b4d-41a4-be55-9fa3da6a9302\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a17de2-669fc94f85-t5cql"
Sep 16 13:12:24.967721 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:24.967698    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c47bfb-9cfc-4105-8f1b-9b2a505e68ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a17de2-54494c4c98-q8gtg\" (UID: \"20c47bfb-9cfc-4105-8f1b-9b2a505e68ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a17de2-54494c4c98-q8gtg"
Sep 16 13:12:24.967849 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:24.967828    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf4ed91e-8b4d-41a4-be55-9fa3da6a9302-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a17de2-669fc94f85-t5cql\" (UID: \"bf4ed91e-8b4d-41a4-be55-9fa3da6a9302\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a17de2-669fc94f85-t5cql"
Sep 16 13:12:24.976677 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:24.976650    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxw79\" (UniqueName: \"kubernetes.io/projected/bf4ed91e-8b4d-41a4-be55-9fa3da6a9302-kube-api-access-cxw79\") pod \"payload-processing-e2e-ipp-a-a17de2-669fc94f85-t5cql\" (UID: \"bf4ed91e-8b4d-41a4-be55-9fa3da6a9302\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a17de2-669fc94f85-t5cql"
Sep 16 13:12:24.977080 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:24.977059    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rtvp\" (UniqueName: \"kubernetes.io/projected/20c47bfb-9cfc-4105-8f1b-9b2a505e68ec-kube-api-access-6rtvp\") pod \"payload-pre-processing-e2e-ipp-a-a17de2-54494c4c98-q8gtg\" (UID: \"20c47bfb-9cfc-4105-8f1b-9b2a505e68ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a17de2-54494c4c98-q8gtg"
Sep 16 13:12:25.075448 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:25.075363    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a17de2-669fc94f85-t5cql"
Sep 16 13:12:25.123716 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:25.123678    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a17de2-54494c4c98-q8gtg"
Sep 16 13:12:25.351921 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:25.351898    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a17de2-669fc94f85-t5cql"]
Sep 16 13:12:25.353602 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:12:25.353573    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf4ed91e_8b4d_41a4_be55_9fa3da6a9302.slice/crio-91dff7dd1ea244d1f65bb8009926a822db29178824e1d77368a7389b68e0b54f WatchSource:0}: Error finding container 91dff7dd1ea244d1f65bb8009926a822db29178824e1d77368a7389b68e0b54f: Status 404 returned error can't find the container with id 91dff7dd1ea244d1f65bb8009926a822db29178824e1d77368a7389b68e0b54f
Sep 16 13:12:25.412113 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:25.412091    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a17de2-54494c4c98-q8gtg"]
Sep 16 13:12:25.415067 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:12:25.415012    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20c47bfb_9cfc_4105_8f1b_9b2a505e68ec.slice/crio-358c296a5c4845fe7bbdbd04e0a7ce30f8e33d4b8243c499105138a94e2a9919 WatchSource:0}: Error finding container 358c296a5c4845fe7bbdbd04e0a7ce30f8e33d4b8243c499105138a94e2a9919: Status 404 returned error can't find the container with id 358c296a5c4845fe7bbdbd04e0a7ce30f8e33d4b8243c499105138a94e2a9919
Sep 16 13:12:25.494118 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:25.494073    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a17de2-54494c4c98-q8gtg" event={"ID":"20c47bfb-9cfc-4105-8f1b-9b2a505e68ec","Type":"ContainerStarted","Data":"42a8ecb0c4db5ec663a15675f27306ca8573344b113bf21e6e714586f15d048e"}
Sep 16 13:12:25.494301 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:25.494162    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a17de2-54494c4c98-q8gtg" event={"ID":"20c47bfb-9cfc-4105-8f1b-9b2a505e68ec","Type":"ContainerStarted","Data":"358c296a5c4845fe7bbdbd04e0a7ce30f8e33d4b8243c499105138a94e2a9919"}
Sep 16 13:12:25.494301 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:25.494291    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a17de2-54494c4c98-q8gtg"
Sep 16 13:12:25.496348 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:25.496321    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a17de2-669fc94f85-t5cql" event={"ID":"bf4ed91e-8b4d-41a4-be55-9fa3da6a9302","Type":"ContainerStarted","Data":"ec9d30f46fea38d4e3c5cb7dd3cd1f8ee02c8e85f52786a57cea89de80f9d3e0"}
Sep 16 13:12:25.496467 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:25.496367    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a17de2-669fc94f85-t5cql" event={"ID":"bf4ed91e-8b4d-41a4-be55-9fa3da6a9302","Type":"ContainerStarted","Data":"91dff7dd1ea244d1f65bb8009926a822db29178824e1d77368a7389b68e0b54f"}
Sep 16 13:12:25.496467 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:25.496404    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a17de2-669fc94f85-t5cql"
Sep 16 13:12:25.512635 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:25.512574    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a17de2-54494c4c98-q8gtg" podStartSLOduration=1.512556494 podStartE2EDuration="1.512556494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:12:25.512496553 +0000 UTC m=+2051.959729193" watchObservedRunningTime="2026-09-16 13:12:25.512556494 +0000 UTC m=+2051.959789140"
Sep 16 13:12:25.534265 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:25.534210    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a17de2-669fc94f85-t5cql" podStartSLOduration=1.534190895 podStartE2EDuration="1.534190895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:12:25.532193449 +0000 UTC m=+2051.979426095" watchObservedRunningTime="2026-09-16 13:12:25.534190895 +0000 UTC m=+2051.981423582"
Sep 16 13:12:33.858453 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:33.858409    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-e344f9/rate-test-model-e344f9-kserve-67455fcdcf-7cpq2" podUID="c48ef941-6fe7-45a5-8ac7-54852d99643f" containerName="main" containerID="cri-o://53197c0c10f377d9b1574690d4615d02923c7756362c45f1634ce12cf7c40c7d" gracePeriod=45
Sep 16 13:12:34.115658 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:34.115588    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e344f9/rate-test-model-e344f9-kserve-67455fcdcf-7cpq2"
Sep 16 13:12:34.265699 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:34.265666    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c48ef941-6fe7-45a5-8ac7-54852d99643f-tls-certs\") pod \"c48ef941-6fe7-45a5-8ac7-54852d99643f\" (UID: \"c48ef941-6fe7-45a5-8ac7-54852d99643f\") "
Sep 16 13:12:34.265853 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:34.265709    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c48ef941-6fe7-45a5-8ac7-54852d99643f-dshm\") pod \"c48ef941-6fe7-45a5-8ac7-54852d99643f\" (UID: \"c48ef941-6fe7-45a5-8ac7-54852d99643f\") "
Sep 16 13:12:34.265853 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:34.265725    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c48ef941-6fe7-45a5-8ac7-54852d99643f-home\") pod \"c48ef941-6fe7-45a5-8ac7-54852d99643f\" (UID: \"c48ef941-6fe7-45a5-8ac7-54852d99643f\") "
Sep 16 13:12:34.265853 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:34.265754    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c48ef941-6fe7-45a5-8ac7-54852d99643f-tmp-dir\") pod \"c48ef941-6fe7-45a5-8ac7-54852d99643f\" (UID: \"c48ef941-6fe7-45a5-8ac7-54852d99643f\") "
Sep 16 13:12:34.265853 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:34.265789    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c48ef941-6fe7-45a5-8ac7-54852d99643f-model-cache\") pod \"c48ef941-6fe7-45a5-8ac7-54852d99643f\" (UID: \"c48ef941-6fe7-45a5-8ac7-54852d99643f\") "
Sep 16 13:12:34.266099 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:34.265857    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dnsh\" (UniqueName: \"kubernetes.io/projected/c48ef941-6fe7-45a5-8ac7-54852d99643f-kube-api-access-2dnsh\") pod \"c48ef941-6fe7-45a5-8ac7-54852d99643f\" (UID: \"c48ef941-6fe7-45a5-8ac7-54852d99643f\") "
Sep 16 13:12:34.266099 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:34.266016    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c48ef941-6fe7-45a5-8ac7-54852d99643f-home" (OuterVolumeSpecName: "home") pod "c48ef941-6fe7-45a5-8ac7-54852d99643f" (UID: "c48ef941-6fe7-45a5-8ac7-54852d99643f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:12:34.266099 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:34.266081    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c48ef941-6fe7-45a5-8ac7-54852d99643f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c48ef941-6fe7-45a5-8ac7-54852d99643f" (UID: "c48ef941-6fe7-45a5-8ac7-54852d99643f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:12:34.266266 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:34.266104    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c48ef941-6fe7-45a5-8ac7-54852d99643f-model-cache" (OuterVolumeSpecName: "model-cache") pod "c48ef941-6fe7-45a5-8ac7-54852d99643f" (UID: "c48ef941-6fe7-45a5-8ac7-54852d99643f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:12:34.266312 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:34.266265    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c48ef941-6fe7-45a5-8ac7-54852d99643f-home\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:12:34.266312 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:34.266279    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c48ef941-6fe7-45a5-8ac7-54852d99643f-tmp-dir\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:12:34.266312 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:34.266288    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c48ef941-6fe7-45a5-8ac7-54852d99643f-model-cache\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:12:34.268136 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:34.268108    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c48ef941-6fe7-45a5-8ac7-54852d99643f-kube-api-access-2dnsh" (OuterVolumeSpecName: "kube-api-access-2dnsh") pod "c48ef941-6fe7-45a5-8ac7-54852d99643f" (UID: "c48ef941-6fe7-45a5-8ac7-54852d99643f"). InnerVolumeSpecName "kube-api-access-2dnsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:12:34.268244 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:34.268139    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c48ef941-6fe7-45a5-8ac7-54852d99643f-dshm" (OuterVolumeSpecName: "dshm") pod "c48ef941-6fe7-45a5-8ac7-54852d99643f" (UID: "c48ef941-6fe7-45a5-8ac7-54852d99643f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:12:34.268244 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:34.268174    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c48ef941-6fe7-45a5-8ac7-54852d99643f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c48ef941-6fe7-45a5-8ac7-54852d99643f" (UID: "c48ef941-6fe7-45a5-8ac7-54852d99643f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:12:34.367774 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:34.367685    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dnsh\" (UniqueName: \"kubernetes.io/projected/c48ef941-6fe7-45a5-8ac7-54852d99643f-kube-api-access-2dnsh\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:12:34.367774 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:34.367713    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c48ef941-6fe7-45a5-8ac7-54852d99643f-tls-certs\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:12:34.367774 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:34.367722    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c48ef941-6fe7-45a5-8ac7-54852d99643f-dshm\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:12:34.548822 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:34.548786    2577 generic.go:356] "Generic (PLEG): container finished" podID="c48ef941-6fe7-45a5-8ac7-54852d99643f" containerID="53197c0c10f377d9b1574690d4615d02923c7756362c45f1634ce12cf7c40c7d" exitCode=0
Sep 16 13:12:34.549022 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:34.548837    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e344f9/rate-test-model-e344f9-kserve-67455fcdcf-7cpq2"
Sep 16 13:12:34.549022 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:34.548875    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e344f9/rate-test-model-e344f9-kserve-67455fcdcf-7cpq2" event={"ID":"c48ef941-6fe7-45a5-8ac7-54852d99643f","Type":"ContainerDied","Data":"53197c0c10f377d9b1574690d4615d02923c7756362c45f1634ce12cf7c40c7d"}
Sep 16 13:12:34.549022 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:34.548927    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e344f9/rate-test-model-e344f9-kserve-67455fcdcf-7cpq2" event={"ID":"c48ef941-6fe7-45a5-8ac7-54852d99643f","Type":"ContainerDied","Data":"e6d2b5369156924952ed83179260187bc003326c5feb096de788cada956877e2"}
Sep 16 13:12:34.549022 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:34.548949    2577 scope.go:117] "RemoveContainer" containerID="53197c0c10f377d9b1574690d4615d02923c7756362c45f1634ce12cf7c40c7d"
Sep 16 13:12:34.558740 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:34.558725    2577 scope.go:117] "RemoveContainer" containerID="53197c0c10f377d9b1574690d4615d02923c7756362c45f1634ce12cf7c40c7d"
Sep 16 13:12:34.559006 ip-10-0-133-190 kubenswrapper[2577]: E0916 13:12:34.558989    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53197c0c10f377d9b1574690d4615d02923c7756362c45f1634ce12cf7c40c7d\": container with ID starting with 53197c0c10f377d9b1574690d4615d02923c7756362c45f1634ce12cf7c40c7d not found: ID does not exist" containerID="53197c0c10f377d9b1574690d4615d02923c7756362c45f1634ce12cf7c40c7d"
Sep 16 13:12:34.559066 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:34.559017    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53197c0c10f377d9b1574690d4615d02923c7756362c45f1634ce12cf7c40c7d"} err="failed to get container status \"53197c0c10f377d9b1574690d4615d02923c7756362c45f1634ce12cf7c40c7d\": rpc error: code = NotFound desc = could not find container \"53197c0c10f377d9b1574690d4615d02923c7756362c45f1634ce12cf7c40c7d\": container with ID starting with 53197c0c10f377d9b1574690d4615d02923c7756362c45f1634ce12cf7c40c7d not found: ID does not exist"
Sep 16 13:12:34.570365 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:34.570342    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e344f9/rate-test-model-e344f9-kserve-67455fcdcf-7cpq2"]
Sep 16 13:12:34.574339 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:34.574318    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e344f9/rate-test-model-e344f9-kserve-67455fcdcf-7cpq2"]
Sep 16 13:12:36.055077 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:36.054984    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c48ef941-6fe7-45a5-8ac7-54852d99643f" path="/var/lib/kubelet/pods/c48ef941-6fe7-45a5-8ac7-54852d99643f/volumes"
Sep 16 13:12:36.506712 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:36.506683    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a17de2-669fc94f85-t5cql"
Sep 16 13:12:36.506977 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:36.506957    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a17de2-54494c4c98-q8gtg"
Sep 16 13:12:48.518341 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:48.518307    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"]
Sep 16 13:12:48.519347 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:48.519322    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c48ef941-6fe7-45a5-8ac7-54852d99643f" containerName="main"
Sep 16 13:12:48.519347 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:48.519349    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c48ef941-6fe7-45a5-8ac7-54852d99643f" containerName="main"
Sep 16 13:12:48.519530 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:48.519450    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c48ef941-6fe7-45a5-8ac7-54852d99643f" containerName="main"
Sep 16 13:12:48.522596 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:48.522576    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"
Sep 16 13:12:48.525540 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:48.525519    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-8d2eac-openshift-default-dockercfg-7qwj4\""
Sep 16 13:12:48.533644 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:48.533619    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"]
Sep 16 13:12:48.705471 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:48.705440    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47140d4f-0539-4afb-a426-d9df88f312e5-workload-certs\") pod \"e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc\" (UID: \"47140d4f-0539-4afb-a426-d9df88f312e5\") " pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"
Sep 16 13:12:48.705471 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:48.705480    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47140d4f-0539-4afb-a426-d9df88f312e5-istio-podinfo\") pod \"e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc\" (UID: \"47140d4f-0539-4afb-a426-d9df88f312e5\") " pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"
Sep 16 13:12:48.705717 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:48.705501    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47140d4f-0539-4afb-a426-d9df88f312e5-istio-data\") pod \"e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc\" (UID: \"47140d4f-0539-4afb-a426-d9df88f312e5\") " pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"
Sep 16 13:12:48.705717 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:48.705599    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47140d4f-0539-4afb-a426-d9df88f312e5-istiod-ca-cert\") pod \"e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc\" (UID: \"47140d4f-0539-4afb-a426-d9df88f312e5\") " pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"
Sep 16 13:12:48.705717 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:48.705650    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47140d4f-0539-4afb-a426-d9df88f312e5-istio-envoy\") pod \"e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc\" (UID: \"47140d4f-0539-4afb-a426-d9df88f312e5\") " pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"
Sep 16 13:12:48.705717 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:48.705675    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47140d4f-0539-4afb-a426-d9df88f312e5-credential-socket\") pod \"e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc\" (UID: \"47140d4f-0539-4afb-a426-d9df88f312e5\") " pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"
Sep 16 13:12:48.705717 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:48.705694    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47140d4f-0539-4afb-a426-d9df88f312e5-workload-socket\") pod \"e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc\" (UID: \"47140d4f-0539-4afb-a426-d9df88f312e5\") " pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"
Sep 16 13:12:48.705955 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:48.705794    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs2jx\" (UniqueName: \"kubernetes.io/projected/47140d4f-0539-4afb-a426-d9df88f312e5-kube-api-access-qs2jx\") pod \"e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc\" (UID: \"47140d4f-0539-4afb-a426-d9df88f312e5\") " pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"
Sep 16 13:12:48.705955 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:48.705841    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47140d4f-0539-4afb-a426-d9df88f312e5-istio-token\") pod \"e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc\" (UID: \"47140d4f-0539-4afb-a426-d9df88f312e5\") " pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"
Sep 16 13:12:48.807145 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:48.807060    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47140d4f-0539-4afb-a426-d9df88f312e5-istiod-ca-cert\") pod \"e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc\" (UID: \"47140d4f-0539-4afb-a426-d9df88f312e5\") " pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"
Sep 16 13:12:48.807145 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:48.807121    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47140d4f-0539-4afb-a426-d9df88f312e5-istio-envoy\") pod \"e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc\" (UID: \"47140d4f-0539-4afb-a426-d9df88f312e5\") " pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"
Sep 16 13:12:48.807145 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:48.807140    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47140d4f-0539-4afb-a426-d9df88f312e5-credential-socket\") pod \"e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc\" (UID: \"47140d4f-0539-4afb-a426-d9df88f312e5\") " pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"
Sep 16 13:12:48.807414 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:48.807158    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47140d4f-0539-4afb-a426-d9df88f312e5-workload-socket\") pod \"e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc\" (UID: \"47140d4f-0539-4afb-a426-d9df88f312e5\") " pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"
Sep 16 13:12:48.807414 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:48.807339    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs2jx\" (UniqueName: \"kubernetes.io/projected/47140d4f-0539-4afb-a426-d9df88f312e5-kube-api-access-qs2jx\") pod \"e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc\" (UID: \"47140d4f-0539-4afb-a426-d9df88f312e5\") " pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"
Sep 16 13:12:48.807414 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:48.807392    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47140d4f-0539-4afb-a426-d9df88f312e5-istio-token\") pod \"e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc\" (UID: \"47140d4f-0539-4afb-a426-d9df88f312e5\") " pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"
Sep 16 13:12:48.807575 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:48.807448    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47140d4f-0539-4afb-a426-d9df88f312e5-workload-certs\") pod \"e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc\" (UID: \"47140d4f-0539-4afb-a426-d9df88f312e5\") " pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"
Sep 16 13:12:48.807575 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:48.807483    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47140d4f-0539-4afb-a426-d9df88f312e5-istio-podinfo\") pod \"e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc\" (UID: \"47140d4f-0539-4afb-a426-d9df88f312e5\") " pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"
Sep 16 13:12:48.807575 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:48.807510    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47140d4f-0539-4afb-a426-d9df88f312e5-istio-data\") pod \"e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc\" (UID: \"47140d4f-0539-4afb-a426-d9df88f312e5\") " pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"
Sep 16 13:12:48.807739 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:48.807573    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47140d4f-0539-4afb-a426-d9df88f312e5-credential-socket\") pod \"e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc\" (UID: \"47140d4f-0539-4afb-a426-d9df88f312e5\") " pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"
Sep 16 13:12:48.807739 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:48.807633    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47140d4f-0539-4afb-a426-d9df88f312e5-workload-socket\") pod \"e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc\" (UID: \"47140d4f-0539-4afb-a426-d9df88f312e5\") " pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"
Sep 16 13:12:48.807841 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:48.807812    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47140d4f-0539-4afb-a426-d9df88f312e5-workload-certs\") pod \"e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc\" (UID: \"47140d4f-0539-4afb-a426-d9df88f312e5\") " pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"
Sep 16 13:12:48.807899 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:48.807850    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47140d4f-0539-4afb-a426-d9df88f312e5-istio-data\") pod \"e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc\" (UID: \"47140d4f-0539-4afb-a426-d9df88f312e5\") " pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"
Sep 16 13:12:48.807899 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:48.807862    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47140d4f-0539-4afb-a426-d9df88f312e5-istiod-ca-cert\") pod \"e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc\" (UID: \"47140d4f-0539-4afb-a426-d9df88f312e5\") " pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"
Sep 16 13:12:48.810075 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:48.810053    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47140d4f-0539-4afb-a426-d9df88f312e5-istio-envoy\") pod \"e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc\" (UID: \"47140d4f-0539-4afb-a426-d9df88f312e5\") " pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"
Sep 16 13:12:48.810208 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:48.810192    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47140d4f-0539-4afb-a426-d9df88f312e5-istio-podinfo\") pod \"e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc\" (UID: \"47140d4f-0539-4afb-a426-d9df88f312e5\") " pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"
Sep 16 13:12:48.822587 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:48.822564    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47140d4f-0539-4afb-a426-d9df88f312e5-istio-token\") pod \"e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc\" (UID: \"47140d4f-0539-4afb-a426-d9df88f312e5\") " pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"
Sep 16 13:12:48.823091 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:48.823072    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs2jx\" (UniqueName: \"kubernetes.io/projected/47140d4f-0539-4afb-a426-d9df88f312e5-kube-api-access-qs2jx\") pod \"e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc\" (UID: \"47140d4f-0539-4afb-a426-d9df88f312e5\") " pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"
Sep 16 13:12:48.834975 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:48.834957    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"
Sep 16 13:12:49.118586 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:49.118560    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"]
Sep 16 13:12:49.120422 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:12:49.120394    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47140d4f_0539_4afb_a426_d9df88f312e5.slice/crio-134933c86eaea6908b9e7dcd8543faea5a2fbd20b10d1ebe354d94c756dcb929 WatchSource:0}: Error finding container 134933c86eaea6908b9e7dcd8543faea5a2fbd20b10d1ebe354d94c756dcb929: Status 404 returned error can't find the container with id 134933c86eaea6908b9e7dcd8543faea5a2fbd20b10d1ebe354d94c756dcb929
Sep 16 13:12:49.122598 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:49.122566    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 13:12:49.122674 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:49.122630    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 13:12:49.122674 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:49.122660    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 13:12:49.636282 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:49.636246    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc" event={"ID":"47140d4f-0539-4afb-a426-d9df88f312e5","Type":"ContainerStarted","Data":"1b946ff8ac7a67bd338f5c7b76a4d29518221bde6e65e693d1037a7e7a5259ce"}
Sep 16 13:12:49.636696 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:49.636301    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc" event={"ID":"47140d4f-0539-4afb-a426-d9df88f312e5","Type":"ContainerStarted","Data":"134933c86eaea6908b9e7dcd8543faea5a2fbd20b10d1ebe354d94c756dcb929"}
Sep 16 13:12:49.664049 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:49.658894    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc" podStartSLOduration=1.6588739399999999 podStartE2EDuration="1.65887394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:12:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:12:49.655741228 +0000 UTC m=+2076.102973875" watchObservedRunningTime="2026-09-16 13:12:49.65887394 +0000 UTC m=+2076.106106586"
Sep 16 13:12:49.835540 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:49.835510    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"
Sep 16 13:12:50.841184 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:50.841155    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"
Sep 16 13:12:51.648965 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:51.648930    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"
Sep 16 13:12:51.650086 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:51.650061    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"
Sep 16 13:12:54.512422 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:54.512378    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8d2eac-6988b44654-frd8b"]
Sep 16 13:12:54.516119 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:54.516101    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d2eac-6988b44654-frd8b"
Sep 16 13:12:54.518624 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:54.518600    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8d2eac\""
Sep 16 13:12:54.518763 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:54.518691    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8d2eac-dockercfg-p5qrw\""
Sep 16 13:12:54.538015 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:54.537985    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8d2eac-6988b44654-frd8b"]
Sep 16 13:12:54.557377 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:54.557349    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47vrc\" (UniqueName: \"kubernetes.io/projected/0b2268c5-3eda-4392-8786-d19a808b3182-kube-api-access-47vrc\") pod \"payload-pre-processing-e2e-ipp-b-8d2eac-6988b44654-frd8b\" (UID: \"0b2268c5-3eda-4392-8786-d19a808b3182\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d2eac-6988b44654-frd8b"
Sep 16 13:12:54.557522 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:54.557404    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b2268c5-3eda-4392-8786-d19a808b3182-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8d2eac-6988b44654-frd8b\" (UID: \"0b2268c5-3eda-4392-8786-d19a808b3182\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d2eac-6988b44654-frd8b"
Sep 16 13:12:54.658298 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:54.658262    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47vrc\" (UniqueName: \"kubernetes.io/projected/0b2268c5-3eda-4392-8786-d19a808b3182-kube-api-access-47vrc\") pod \"payload-pre-processing-e2e-ipp-b-8d2eac-6988b44654-frd8b\" (UID: \"0b2268c5-3eda-4392-8786-d19a808b3182\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d2eac-6988b44654-frd8b"
Sep 16 13:12:54.658486 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:54.658326    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b2268c5-3eda-4392-8786-d19a808b3182-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8d2eac-6988b44654-frd8b\" (UID: \"0b2268c5-3eda-4392-8786-d19a808b3182\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d2eac-6988b44654-frd8b"
Sep 16 13:12:54.659041 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:54.659002    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b2268c5-3eda-4392-8786-d19a808b3182-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8d2eac-6988b44654-frd8b\" (UID: \"0b2268c5-3eda-4392-8786-d19a808b3182\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d2eac-6988b44654-frd8b"
Sep 16 13:12:54.666086 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:54.666060    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47vrc\" (UniqueName: \"kubernetes.io/projected/0b2268c5-3eda-4392-8786-d19a808b3182-kube-api-access-47vrc\") pod \"payload-pre-processing-e2e-ipp-b-8d2eac-6988b44654-frd8b\" (UID: \"0b2268c5-3eda-4392-8786-d19a808b3182\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d2eac-6988b44654-frd8b"
Sep 16 13:12:54.837410 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:54.837323    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d2eac-6988b44654-frd8b"
Sep 16 13:12:55.117154 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:55.117129    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8d2eac-6988b44654-frd8b"]
Sep 16 13:12:55.118911 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:12:55.118885    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b2268c5_3eda_4392_8786_d19a808b3182.slice/crio-85a9527ee287118e95e119b8920b1afd4511d86a343fb76f8e9d39d793cfc462 WatchSource:0}: Error finding container 85a9527ee287118e95e119b8920b1afd4511d86a343fb76f8e9d39d793cfc462: Status 404 returned error can't find the container with id 85a9527ee287118e95e119b8920b1afd4511d86a343fb76f8e9d39d793cfc462
Sep 16 13:12:55.671648 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:55.671616    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d2eac-6988b44654-frd8b" event={"ID":"0b2268c5-3eda-4392-8786-d19a808b3182","Type":"ContainerStarted","Data":"ede9c20e9f6ff35ee00f03d8d9fbda80a3219d8ceaa38b1797c23caca60a8eb2"}
Sep 16 13:12:55.672121 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:55.671678    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d2eac-6988b44654-frd8b"
Sep 16 13:12:55.672121 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:55.671700    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d2eac-6988b44654-frd8b" event={"ID":"0b2268c5-3eda-4392-8786-d19a808b3182","Type":"ContainerStarted","Data":"85a9527ee287118e95e119b8920b1afd4511d86a343fb76f8e9d39d793cfc462"}
Sep 16 13:12:55.690340 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:12:55.690298    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d2eac-6988b44654-frd8b" podStartSLOduration=1.690283279 podStartE2EDuration="1.690283279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:12:55.688379344 +0000 UTC m=+2082.135611987" watchObservedRunningTime="2026-09-16 13:12:55.690283279 +0000 UTC m=+2082.137515920"
Sep 16 13:13:06.679182 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:06.679148    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d2eac-6988b44654-frd8b"
Sep 16 13:13:07.690227 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:07.690177    2577 patch_prober.go:28] interesting pod/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 13:13:07.690639 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:07.690279    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc" podUID="47140d4f-0539-4afb-a426-d9df88f312e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 13:13:09.765051 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:09.764994    2577 generic.go:356] "Generic (PLEG): container finished" podID="47140d4f-0539-4afb-a426-d9df88f312e5" containerID="1b946ff8ac7a67bd338f5c7b76a4d29518221bde6e65e693d1037a7e7a5259ce" exitCode=0
Sep 16 13:13:09.765532 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:09.765060    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc" event={"ID":"47140d4f-0539-4afb-a426-d9df88f312e5","Type":"ContainerDied","Data":"1b946ff8ac7a67bd338f5c7b76a4d29518221bde6e65e693d1037a7e7a5259ce"}
Sep 16 13:13:09.765532 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:09.765449    2577 scope.go:117] "RemoveContainer" containerID="1b946ff8ac7a67bd338f5c7b76a4d29518221bde6e65e693d1037a7e7a5259ce"
Sep 16 13:13:09.766514 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:09.766484    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 13:13:09.766570 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:09.766555    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 13:13:09.766606 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:09.766585    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 13:13:10.772654 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:10.772621    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc" event={"ID":"47140d4f-0539-4afb-a426-d9df88f312e5","Type":"ContainerStarted","Data":"f29bc91212e2a68fa2f8cc62acd122a04c1de131ffcb3e918cfe6030d4b944b2"}
Sep 16 13:13:10.836056 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:10.835997    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"
Sep 16 13:13:10.836242 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:10.836219    2577 patch_prober.go:28] interesting pod/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Sep 16 13:13:10.836306 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:10.836285    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc" podUID="47140d4f-0539-4afb-a426-d9df88f312e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Sep 16 13:13:11.835648 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:11.835609    2577 patch_prober.go:28] interesting pod/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Sep 16 13:13:11.836116 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:11.835679    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc" podUID="47140d4f-0539-4afb-a426-d9df88f312e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Sep 16 13:13:12.835861 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:12.835824    2577 patch_prober.go:28] interesting pod/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Sep 16 13:13:12.836281 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:12.835879    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc" podUID="47140d4f-0539-4afb-a426-d9df88f312e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Sep 16 13:13:13.835494 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:13.835456    2577 patch_prober.go:28] interesting pod/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Sep 16 13:13:13.835658 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:13.835516    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc" podUID="47140d4f-0539-4afb-a426-d9df88f312e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Sep 16 13:13:14.230225 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:14.230147    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vbt8d_0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174/console-operator/1.log"
Sep 16 13:13:14.234714 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:14.234696    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vbt8d_0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174/console-operator/1.log"
Sep 16 13:13:14.835386 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:14.835351    2577 patch_prober.go:28] interesting pod/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Sep 16 13:13:14.835574 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:14.835419    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc" podUID="47140d4f-0539-4afb-a426-d9df88f312e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Sep 16 13:13:15.841273 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:15.841236    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"
Sep 16 13:13:15.841770 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:15.841649    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"
Sep 16 13:13:15.842461 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:15.842444    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"
Sep 16 13:13:30.756087 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:30.756049    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-a17de2/ipp-route-a17de2-kserve-6c6d888c47-lpx7c"]
Sep 16 13:13:30.760312 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:30.760228    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a17de2/ipp-route-a17de2-kserve-6c6d888c47-lpx7c"
Sep 16 13:13:30.763466 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:30.763440    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a17de2\"/\"default-dockercfg-lbnch\""
Sep 16 13:13:30.764499 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:30.764485    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a17de2\"/\"openshift-service-ca.crt\""
Sep 16 13:13:30.764650 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:30.764631    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a17de2\"/\"kube-root-ca.crt\""
Sep 16 13:13:30.764825 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:30.764811    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a17de2\"/\"ipp-route-a17de2-kserve-self-signed-certs\""
Sep 16 13:13:30.773879 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:30.773856    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a17de2/ipp-route-a17de2-kserve-6c6d888c47-lpx7c"]
Sep 16 13:13:30.902588 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:30.902542    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59b8b489-8e35-456b-8e40-0d08ec849f6f-tls-certs\") pod \"ipp-route-a17de2-kserve-6c6d888c47-lpx7c\" (UID: \"59b8b489-8e35-456b-8e40-0d08ec849f6f\") " pod="ai-tenant-e2e-ipp-a-a17de2/ipp-route-a17de2-kserve-6c6d888c47-lpx7c"
Sep 16 13:13:30.902767 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:30.902606    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59b8b489-8e35-456b-8e40-0d08ec849f6f-home\") pod \"ipp-route-a17de2-kserve-6c6d888c47-lpx7c\" (UID: \"59b8b489-8e35-456b-8e40-0d08ec849f6f\") " pod="ai-tenant-e2e-ipp-a-a17de2/ipp-route-a17de2-kserve-6c6d888c47-lpx7c"
Sep 16 13:13:30.902767 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:30.902638    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59b8b489-8e35-456b-8e40-0d08ec849f6f-model-cache\") pod \"ipp-route-a17de2-kserve-6c6d888c47-lpx7c\" (UID: \"59b8b489-8e35-456b-8e40-0d08ec849f6f\") " pod="ai-tenant-e2e-ipp-a-a17de2/ipp-route-a17de2-kserve-6c6d888c47-lpx7c"
Sep 16 13:13:30.902767 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:30.902737    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59b8b489-8e35-456b-8e40-0d08ec849f6f-tmp-dir\") pod \"ipp-route-a17de2-kserve-6c6d888c47-lpx7c\" (UID: \"59b8b489-8e35-456b-8e40-0d08ec849f6f\") " pod="ai-tenant-e2e-ipp-a-a17de2/ipp-route-a17de2-kserve-6c6d888c47-lpx7c"
Sep 16 13:13:30.902872 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:30.902782    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59b8b489-8e35-456b-8e40-0d08ec849f6f-dshm\") pod \"ipp-route-a17de2-kserve-6c6d888c47-lpx7c\" (UID: \"59b8b489-8e35-456b-8e40-0d08ec849f6f\") " pod="ai-tenant-e2e-ipp-a-a17de2/ipp-route-a17de2-kserve-6c6d888c47-lpx7c"
Sep 16 13:13:30.902924 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:30.902899    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrzcg\" (UniqueName: \"kubernetes.io/projected/59b8b489-8e35-456b-8e40-0d08ec849f6f-kube-api-access-mrzcg\") pod \"ipp-route-a17de2-kserve-6c6d888c47-lpx7c\" (UID: \"59b8b489-8e35-456b-8e40-0d08ec849f6f\") " pod="ai-tenant-e2e-ipp-a-a17de2/ipp-route-a17de2-kserve-6c6d888c47-lpx7c"
Sep 16 13:13:31.004183 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:31.004151    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59b8b489-8e35-456b-8e40-0d08ec849f6f-tmp-dir\") pod \"ipp-route-a17de2-kserve-6c6d888c47-lpx7c\" (UID: \"59b8b489-8e35-456b-8e40-0d08ec849f6f\") " pod="ai-tenant-e2e-ipp-a-a17de2/ipp-route-a17de2-kserve-6c6d888c47-lpx7c"
Sep 16 13:13:31.004183 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:31.004198    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59b8b489-8e35-456b-8e40-0d08ec849f6f-dshm\") pod \"ipp-route-a17de2-kserve-6c6d888c47-lpx7c\" (UID: \"59b8b489-8e35-456b-8e40-0d08ec849f6f\") " pod="ai-tenant-e2e-ipp-a-a17de2/ipp-route-a17de2-kserve-6c6d888c47-lpx7c"
Sep 16 13:13:31.004446 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:31.004249    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrzcg\" (UniqueName: \"kubernetes.io/projected/59b8b489-8e35-456b-8e40-0d08ec849f6f-kube-api-access-mrzcg\") pod \"ipp-route-a17de2-kserve-6c6d888c47-lpx7c\" (UID: \"59b8b489-8e35-456b-8e40-0d08ec849f6f\") " pod="ai-tenant-e2e-ipp-a-a17de2/ipp-route-a17de2-kserve-6c6d888c47-lpx7c"
Sep 16 13:13:31.004446 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:31.004276    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59b8b489-8e35-456b-8e40-0d08ec849f6f-tls-certs\") pod \"ipp-route-a17de2-kserve-6c6d888c47-lpx7c\" (UID: \"59b8b489-8e35-456b-8e40-0d08ec849f6f\") " pod="ai-tenant-e2e-ipp-a-a17de2/ipp-route-a17de2-kserve-6c6d888c47-lpx7c"
Sep 16 13:13:31.004446 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:31.004303    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59b8b489-8e35-456b-8e40-0d08ec849f6f-home\") pod \"ipp-route-a17de2-kserve-6c6d888c47-lpx7c\" (UID: \"59b8b489-8e35-456b-8e40-0d08ec849f6f\") " pod="ai-tenant-e2e-ipp-a-a17de2/ipp-route-a17de2-kserve-6c6d888c47-lpx7c"
Sep 16 13:13:31.004446 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:31.004328    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59b8b489-8e35-456b-8e40-0d08ec849f6f-model-cache\") pod \"ipp-route-a17de2-kserve-6c6d888c47-lpx7c\" (UID: \"59b8b489-8e35-456b-8e40-0d08ec849f6f\") " pod="ai-tenant-e2e-ipp-a-a17de2/ipp-route-a17de2-kserve-6c6d888c47-lpx7c"
Sep 16 13:13:31.004661 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:31.004616    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59b8b489-8e35-456b-8e40-0d08ec849f6f-tmp-dir\") pod \"ipp-route-a17de2-kserve-6c6d888c47-lpx7c\" (UID: \"59b8b489-8e35-456b-8e40-0d08ec849f6f\") " pod="ai-tenant-e2e-ipp-a-a17de2/ipp-route-a17de2-kserve-6c6d888c47-lpx7c"
Sep 16 13:13:31.004661 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:31.004650    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59b8b489-8e35-456b-8e40-0d08ec849f6f-home\") pod \"ipp-route-a17de2-kserve-6c6d888c47-lpx7c\" (UID: \"59b8b489-8e35-456b-8e40-0d08ec849f6f\") " pod="ai-tenant-e2e-ipp-a-a17de2/ipp-route-a17de2-kserve-6c6d888c47-lpx7c"
Sep 16 13:13:31.004744 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:31.004722    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59b8b489-8e35-456b-8e40-0d08ec849f6f-model-cache\") pod \"ipp-route-a17de2-kserve-6c6d888c47-lpx7c\" (UID: \"59b8b489-8e35-456b-8e40-0d08ec849f6f\") " pod="ai-tenant-e2e-ipp-a-a17de2/ipp-route-a17de2-kserve-6c6d888c47-lpx7c"
Sep 16 13:13:31.006786 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:31.006722    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59b8b489-8e35-456b-8e40-0d08ec849f6f-dshm\") pod \"ipp-route-a17de2-kserve-6c6d888c47-lpx7c\" (UID: \"59b8b489-8e35-456b-8e40-0d08ec849f6f\") " pod="ai-tenant-e2e-ipp-a-a17de2/ipp-route-a17de2-kserve-6c6d888c47-lpx7c"
Sep 16 13:13:31.007216 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:31.007192    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59b8b489-8e35-456b-8e40-0d08ec849f6f-tls-certs\") pod \"ipp-route-a17de2-kserve-6c6d888c47-lpx7c\" (UID: \"59b8b489-8e35-456b-8e40-0d08ec849f6f\") " pod="ai-tenant-e2e-ipp-a-a17de2/ipp-route-a17de2-kserve-6c6d888c47-lpx7c"
Sep 16 13:13:31.016245 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:31.016222    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrzcg\" (UniqueName: \"kubernetes.io/projected/59b8b489-8e35-456b-8e40-0d08ec849f6f-kube-api-access-mrzcg\") pod \"ipp-route-a17de2-kserve-6c6d888c47-lpx7c\" (UID: \"59b8b489-8e35-456b-8e40-0d08ec849f6f\") " pod="ai-tenant-e2e-ipp-a-a17de2/ipp-route-a17de2-kserve-6c6d888c47-lpx7c"
Sep 16 13:13:31.070935 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:31.070905    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a17de2/ipp-route-a17de2-kserve-6c6d888c47-lpx7c"
Sep 16 13:13:31.350264 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:31.350241    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a17de2/ipp-route-a17de2-kserve-6c6d888c47-lpx7c"]
Sep 16 13:13:31.352663 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:13:31.352635    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59b8b489_8e35_456b_8e40_0d08ec849f6f.slice/crio-cf5476678c2abd1f1b69867f30b8c7c78ee1dd772fb9915eb2c3be09b9afcc7a WatchSource:0}: Error finding container cf5476678c2abd1f1b69867f30b8c7c78ee1dd772fb9915eb2c3be09b9afcc7a: Status 404 returned error can't find the container with id cf5476678c2abd1f1b69867f30b8c7c78ee1dd772fb9915eb2c3be09b9afcc7a
Sep 16 13:13:31.919882 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:31.919851    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a17de2/ipp-route-a17de2-kserve-6c6d888c47-lpx7c" event={"ID":"59b8b489-8e35-456b-8e40-0d08ec849f6f","Type":"ContainerStarted","Data":"cfa4f73883646aa6d70bda191c1c73daa015ed8fd85e9ed42051c84d25006730"}
Sep 16 13:13:31.919882 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:31.919890    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a17de2/ipp-route-a17de2-kserve-6c6d888c47-lpx7c" event={"ID":"59b8b489-8e35-456b-8e40-0d08ec849f6f","Type":"ContainerStarted","Data":"cf5476678c2abd1f1b69867f30b8c7c78ee1dd772fb9915eb2c3be09b9afcc7a"}
Sep 16 13:13:31.941183 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:31.941137    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-a17de2/ipp-route-a17de2-kserve-6c6d888c47-lpx7c" podStartSLOduration=1.941122659 podStartE2EDuration="1.941122659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:13:31.939301994 +0000 UTC m=+2118.386534636" watchObservedRunningTime="2026-09-16 13:13:31.941122659 +0000 UTC m=+2118.388355300"
Sep 16 13:13:41.071272 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:41.071230    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-a17de2/ipp-route-a17de2-kserve-6c6d888c47-lpx7c"
Sep 16 13:13:41.071773 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:41.071294    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-a17de2/ipp-route-a17de2-kserve-6c6d888c47-lpx7c"
Sep 16 13:13:41.084019 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:41.083993    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-a17de2/ipp-route-a17de2-kserve-6c6d888c47-lpx7c"
Sep 16 13:13:41.989634 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:13:41.989600    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-a17de2/ipp-route-a17de2-kserve-6c6d888c47-lpx7c"
Sep 16 13:14:02.954456 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:02.954426    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-a17de2-669fc94f85-t5cql_bf4ed91e-8b4d-41a4-be55-9fa3da6a9302/payload-processing/0.log"
Sep 16 13:14:11.765255 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:11.765167    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d20caf-745b94bf5-f5hrb"]
Sep 16 13:14:11.769582 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:11.769551    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d20caf-745b94bf5-f5hrb"
Sep 16 13:14:11.772145 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:11.772127    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-d20caf-dockercfg-hq82h\""
Sep 16 13:14:11.772406 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:11.772384    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-d20caf\""
Sep 16 13:14:11.782360 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:11.782339    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d20caf-745b94bf5-f5hrb"]
Sep 16 13:14:11.913372 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:11.913340    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/989941ce-7b79-4888-9b16-df298367996a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-d20caf-745b94bf5-f5hrb\" (UID: \"989941ce-7b79-4888-9b16-df298367996a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d20caf-745b94bf5-f5hrb"
Sep 16 13:14:11.913558 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:11.913409    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8q22\" (UniqueName: \"kubernetes.io/projected/989941ce-7b79-4888-9b16-df298367996a-kube-api-access-t8q22\") pod \"payload-processing-e2e-ipp-cleanup-d20caf-745b94bf5-f5hrb\" (UID: \"989941ce-7b79-4888-9b16-df298367996a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d20caf-745b94bf5-f5hrb"
Sep 16 13:14:12.014363 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:12.014326    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/989941ce-7b79-4888-9b16-df298367996a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-d20caf-745b94bf5-f5hrb\" (UID: \"989941ce-7b79-4888-9b16-df298367996a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d20caf-745b94bf5-f5hrb"
Sep 16 13:14:12.014545 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:12.014393    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8q22\" (UniqueName: \"kubernetes.io/projected/989941ce-7b79-4888-9b16-df298367996a-kube-api-access-t8q22\") pod \"payload-processing-e2e-ipp-cleanup-d20caf-745b94bf5-f5hrb\" (UID: \"989941ce-7b79-4888-9b16-df298367996a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d20caf-745b94bf5-f5hrb"
Sep 16 13:14:12.015102 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:12.015077    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/989941ce-7b79-4888-9b16-df298367996a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-d20caf-745b94bf5-f5hrb\" (UID: \"989941ce-7b79-4888-9b16-df298367996a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d20caf-745b94bf5-f5hrb"
Sep 16 13:14:12.023350 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:12.023299    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8q22\" (UniqueName: \"kubernetes.io/projected/989941ce-7b79-4888-9b16-df298367996a-kube-api-access-t8q22\") pod \"payload-processing-e2e-ipp-cleanup-d20caf-745b94bf5-f5hrb\" (UID: \"989941ce-7b79-4888-9b16-df298367996a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d20caf-745b94bf5-f5hrb"
Sep 16 13:14:12.080831 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:12.080787    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d20caf-745b94bf5-f5hrb"
Sep 16 13:14:12.358343 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:12.358315    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d20caf-745b94bf5-f5hrb"]
Sep 16 13:14:12.359881 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:14:12.359856    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod989941ce_7b79_4888_9b16_df298367996a.slice/crio-ff0321b655bf48b1d5bc9c2a76f79cc38b63d0ef782074cf7647c77c7ccc466b WatchSource:0}: Error finding container ff0321b655bf48b1d5bc9c2a76f79cc38b63d0ef782074cf7647c77c7ccc466b: Status 404 returned error can't find the container with id ff0321b655bf48b1d5bc9c2a76f79cc38b63d0ef782074cf7647c77c7ccc466b
Sep 16 13:14:13.158936 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:13.158899    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d20caf-745b94bf5-f5hrb" event={"ID":"989941ce-7b79-4888-9b16-df298367996a","Type":"ContainerStarted","Data":"8f2ec7c4be7e42aa84356d3b85c1d8654328c195c8219ac31c896888ea8a4214"}
Sep 16 13:14:13.159386 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:13.158962    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d20caf-745b94bf5-f5hrb" event={"ID":"989941ce-7b79-4888-9b16-df298367996a","Type":"ContainerStarted","Data":"ff0321b655bf48b1d5bc9c2a76f79cc38b63d0ef782074cf7647c77c7ccc466b"}
Sep 16 13:14:13.159386 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:13.159037    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d20caf-745b94bf5-f5hrb"
Sep 16 13:14:13.182251 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:13.182198    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d20caf-745b94bf5-f5hrb" podStartSLOduration=2.1821825 podStartE2EDuration="2.1821825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:14:13.180659522 +0000 UTC m=+2159.627892167" watchObservedRunningTime="2026-09-16 13:14:13.1821825 +0000 UTC m=+2159.629415142"
Sep 16 13:14:24.167213 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:24.167184    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d20caf-745b94bf5-f5hrb"
Sep 16 13:14:38.057638 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:38.057597    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d20caf-w588f"]
Sep 16 13:14:38.061320 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:38.061291    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d20caf-w588f"
Sep 16 13:14:38.064173 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:38.064153    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-64zph\""
Sep 16 13:14:38.064292 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:38.064153    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 13:14:38.070980 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:38.070951    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d20caf-w588f"]
Sep 16 13:14:38.169566 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:38.167346    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/934360ab-3048-4f1d-ae38-2871681a4dd1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-d20caf-w588f\" (UID: \"934360ab-3048-4f1d-ae38-2871681a4dd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d20caf-w588f"
Sep 16 13:14:38.268346 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:38.268313    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/934360ab-3048-4f1d-ae38-2871681a4dd1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-d20caf-w588f\" (UID: \"934360ab-3048-4f1d-ae38-2871681a4dd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d20caf-w588f"
Sep 16 13:14:38.268911 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:38.268889    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/934360ab-3048-4f1d-ae38-2871681a4dd1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-d20caf-w588f\" (UID: \"934360ab-3048-4f1d-ae38-2871681a4dd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d20caf-w588f"
Sep 16 13:14:38.372139 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:38.372069    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d20caf-w588f"
Sep 16 13:14:38.649934 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:38.649909    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d20caf-w588f"]
Sep 16 13:14:38.652947 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:14:38.652910    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod934360ab_3048_4f1d_ae38_2871681a4dd1.slice/crio-0c257e09ecc65174725101c3fc137a737b30f4e79448311abfd338b0e9dd8c01 WatchSource:0}: Error finding container 0c257e09ecc65174725101c3fc137a737b30f4e79448311abfd338b0e9dd8c01: Status 404 returned error can't find the container with id 0c257e09ecc65174725101c3fc137a737b30f4e79448311abfd338b0e9dd8c01
Sep 16 13:14:39.322368 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:39.322338    2577 generic.go:356] "Generic (PLEG): container finished" podID="934360ab-3048-4f1d-ae38-2871681a4dd1" containerID="7996270a2088626ab38d47793892fc34dcfe6a8a0f406072817d6a8d417ba8d8" exitCode=0
Sep 16 13:14:39.322683 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:39.322394    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d20caf-w588f" event={"ID":"934360ab-3048-4f1d-ae38-2871681a4dd1","Type":"ContainerDied","Data":"7996270a2088626ab38d47793892fc34dcfe6a8a0f406072817d6a8d417ba8d8"}
Sep 16 13:14:39.322683 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:39.322437    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d20caf-w588f" event={"ID":"934360ab-3048-4f1d-ae38-2871681a4dd1","Type":"ContainerStarted","Data":"0c257e09ecc65174725101c3fc137a737b30f4e79448311abfd338b0e9dd8c01"}
Sep 16 13:14:40.471592 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:40.471570    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d20caf-w588f"
Sep 16 13:14:40.591876 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:40.591793    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/934360ab-3048-4f1d-ae38-2871681a4dd1-maas-api-service-ca\") pod \"934360ab-3048-4f1d-ae38-2871681a4dd1\" (UID: \"934360ab-3048-4f1d-ae38-2871681a4dd1\") "
Sep 16 13:14:40.592214 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:40.592190    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/934360ab-3048-4f1d-ae38-2871681a4dd1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "934360ab-3048-4f1d-ae38-2871681a4dd1" (UID: "934360ab-3048-4f1d-ae38-2871681a4dd1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:14:40.692531 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:40.692492    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/934360ab-3048-4f1d-ae38-2871681a4dd1-maas-api-service-ca\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:14:41.340627 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:41.340595    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d20caf-w588f"
Sep 16 13:14:41.340796 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:41.340595    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d20caf-w588f" event={"ID":"934360ab-3048-4f1d-ae38-2871681a4dd1","Type":"ContainerDied","Data":"0c257e09ecc65174725101c3fc137a737b30f4e79448311abfd338b0e9dd8c01"}
Sep 16 13:14:41.340796 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:41.340748    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c257e09ecc65174725101c3fc137a737b30f4e79448311abfd338b0e9dd8c01"
Sep 16 13:14:49.057112 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:49.057079    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d20caf-745b94bf5-f5hrb"]
Sep 16 13:14:49.057587 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:49.057347    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d20caf-745b94bf5-f5hrb" podUID="989941ce-7b79-4888-9b16-df298367996a" containerName="payload-processing" containerID="cri-o://8f2ec7c4be7e42aa84356d3b85c1d8654328c195c8219ac31c896888ea8a4214" gracePeriod=30
Sep 16 13:14:49.213195 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:49.213155    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d20caf-w588f"]
Sep 16 13:14:49.217238 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:49.217207    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d20caf-w588f"]
Sep 16 13:14:49.329398 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:49.329374    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d20caf-745b94bf5-f5hrb"
Sep 16 13:14:49.389020 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:49.388984    2577 generic.go:356] "Generic (PLEG): container finished" podID="989941ce-7b79-4888-9b16-df298367996a" containerID="8f2ec7c4be7e42aa84356d3b85c1d8654328c195c8219ac31c896888ea8a4214" exitCode=0
Sep 16 13:14:49.389187 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:49.389063    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d20caf-745b94bf5-f5hrb"
Sep 16 13:14:49.389187 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:49.389079    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d20caf-745b94bf5-f5hrb" event={"ID":"989941ce-7b79-4888-9b16-df298367996a","Type":"ContainerDied","Data":"8f2ec7c4be7e42aa84356d3b85c1d8654328c195c8219ac31c896888ea8a4214"}
Sep 16 13:14:49.389187 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:49.389145    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d20caf-745b94bf5-f5hrb" event={"ID":"989941ce-7b79-4888-9b16-df298367996a","Type":"ContainerDied","Data":"ff0321b655bf48b1d5bc9c2a76f79cc38b63d0ef782074cf7647c77c7ccc466b"}
Sep 16 13:14:49.389187 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:49.389175    2577 scope.go:117] "RemoveContainer" containerID="8f2ec7c4be7e42aa84356d3b85c1d8654328c195c8219ac31c896888ea8a4214"
Sep 16 13:14:49.399797 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:49.399775    2577 scope.go:117] "RemoveContainer" containerID="8f2ec7c4be7e42aa84356d3b85c1d8654328c195c8219ac31c896888ea8a4214"
Sep 16 13:14:49.400161 ip-10-0-133-190 kubenswrapper[2577]: E0916 13:14:49.400124    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f2ec7c4be7e42aa84356d3b85c1d8654328c195c8219ac31c896888ea8a4214\": container with ID starting with 8f2ec7c4be7e42aa84356d3b85c1d8654328c195c8219ac31c896888ea8a4214 not found: ID does not exist" containerID="8f2ec7c4be7e42aa84356d3b85c1d8654328c195c8219ac31c896888ea8a4214"
Sep 16 13:14:49.400286 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:49.400162    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f2ec7c4be7e42aa84356d3b85c1d8654328c195c8219ac31c896888ea8a4214"} err="failed to get container status \"8f2ec7c4be7e42aa84356d3b85c1d8654328c195c8219ac31c896888ea8a4214\": rpc error: code = NotFound desc = could not find container \"8f2ec7c4be7e42aa84356d3b85c1d8654328c195c8219ac31c896888ea8a4214\": container with ID starting with 8f2ec7c4be7e42aa84356d3b85c1d8654328c195c8219ac31c896888ea8a4214 not found: ID does not exist"
Sep 16 13:14:49.487585 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:49.487559    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/989941ce-7b79-4888-9b16-df298367996a-plugins-config-volume\") pod \"989941ce-7b79-4888-9b16-df298367996a\" (UID: \"989941ce-7b79-4888-9b16-df298367996a\") "
Sep 16 13:14:49.487774 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:49.487618    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8q22\" (UniqueName: \"kubernetes.io/projected/989941ce-7b79-4888-9b16-df298367996a-kube-api-access-t8q22\") pod \"989941ce-7b79-4888-9b16-df298367996a\" (UID: \"989941ce-7b79-4888-9b16-df298367996a\") "
Sep 16 13:14:49.487998 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:49.487973    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/989941ce-7b79-4888-9b16-df298367996a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "989941ce-7b79-4888-9b16-df298367996a" (UID: "989941ce-7b79-4888-9b16-df298367996a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:14:49.489958 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:49.489934    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/989941ce-7b79-4888-9b16-df298367996a-kube-api-access-t8q22" (OuterVolumeSpecName: "kube-api-access-t8q22") pod "989941ce-7b79-4888-9b16-df298367996a" (UID: "989941ce-7b79-4888-9b16-df298367996a"). InnerVolumeSpecName "kube-api-access-t8q22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:14:49.589374 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:49.589275    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8q22\" (UniqueName: \"kubernetes.io/projected/989941ce-7b79-4888-9b16-df298367996a-kube-api-access-t8q22\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:14:49.589374 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:49.589328    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/989941ce-7b79-4888-9b16-df298367996a-plugins-config-volume\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:14:49.724638 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:49.724599    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d20caf-745b94bf5-f5hrb"]
Sep 16 13:14:49.736167 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:49.736142    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d20caf-745b94bf5-f5hrb"]
Sep 16 13:14:50.055881 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:50.055851    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="934360ab-3048-4f1d-ae38-2871681a4dd1" path="/var/lib/kubelet/pods/934360ab-3048-4f1d-ae38-2871681a4dd1/volumes"
Sep 16 13:14:50.056303 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:50.056284    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="989941ce-7b79-4888-9b16-df298367996a" path="/var/lib/kubelet/pods/989941ce-7b79-4888-9b16-df298367996a/volumes"
Sep 16 13:14:59.703046 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:59.702997    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a17de2-lmgzx"]
Sep 16 13:14:59.703496 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:59.703480    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="934360ab-3048-4f1d-ae38-2871681a4dd1" containerName="revoke-keys"
Sep 16 13:14:59.703496 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:59.703496    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="934360ab-3048-4f1d-ae38-2871681a4dd1" containerName="revoke-keys"
Sep 16 13:14:59.703582 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:59.703514    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="989941ce-7b79-4888-9b16-df298367996a" containerName="payload-processing"
Sep 16 13:14:59.703582 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:59.703521    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="989941ce-7b79-4888-9b16-df298367996a" containerName="payload-processing"
Sep 16 13:14:59.703645 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:59.703589    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="989941ce-7b79-4888-9b16-df298367996a" containerName="payload-processing"
Sep 16 13:14:59.703645 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:59.703597    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="934360ab-3048-4f1d-ae38-2871681a4dd1" containerName="revoke-keys"
Sep 16 13:14:59.707988 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:59.707971    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a17de2-lmgzx"
Sep 16 13:14:59.710534 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:59.710513    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 13:14:59.710534 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:59.710527    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-64zph\""
Sep 16 13:14:59.715981 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:59.715954    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a17de2-lmgzx"]
Sep 16 13:14:59.789199 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:59.789167    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/557d388b-b04b-4d34-a6a6-7aec9824fd3f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a17de2-lmgzx\" (UID: \"557d388b-b04b-4d34-a6a6-7aec9824fd3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a17de2-lmgzx"
Sep 16 13:14:59.890504 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:59.890469    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/557d388b-b04b-4d34-a6a6-7aec9824fd3f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a17de2-lmgzx\" (UID: \"557d388b-b04b-4d34-a6a6-7aec9824fd3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a17de2-lmgzx"
Sep 16 13:14:59.891197 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:14:59.891173    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/557d388b-b04b-4d34-a6a6-7aec9824fd3f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a17de2-lmgzx\" (UID: \"557d388b-b04b-4d34-a6a6-7aec9824fd3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a17de2-lmgzx"
Sep 16 13:15:00.020230 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:00.020201    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a17de2-lmgzx"
Sep 16 13:15:00.141467 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:00.141419    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-8d2eac-29826075-89wgf"]
Sep 16 13:15:00.145440 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:00.145305    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-8d2eac-29826075-89wgf"
Sep 16 13:15:00.148385 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:00.148361    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e344f9-29826075-ftnsd"]
Sep 16 13:15:00.152466 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:00.152359    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826075-mtpck"]
Sep 16 13:15:00.153056 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:00.152686    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e344f9-29826075-ftnsd"
Sep 16 13:15:00.163317 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:00.163295    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-8d2eac-29826075-89wgf"]
Sep 16 13:15:00.163568 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:00.163553    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826075-mtpck"]
Sep 16 13:15:00.163693 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:00.163681    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e344f9-29826075-ftnsd"]
Sep 16 13:15:00.163823 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:00.163763    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-8d2eac-29826075-89wgf"
Sep 16 13:15:00.163977 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:00.163867    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826075-mtpck"
Sep 16 13:15:00.174244 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:00.174225    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e344f9-29826075-ftnsd"
Sep 16 13:15:00.208471 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:00.208392    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826075-mtpck"
Sep 16 13:15:00.298318 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:00.298286    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a17de2-lmgzx"]
Sep 16 13:15:00.301882 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:15:00.301853    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod557d388b_b04b_4d34_a6a6_7aec9824fd3f.slice/crio-312836c7b31acab60248286a85bbdbbc7874165953c1fb3cc5213714227675c5 WatchSource:0}: Error finding container 312836c7b31acab60248286a85bbdbbc7874165953c1fb3cc5213714227675c5: Status 404 returned error can't find the container with id 312836c7b31acab60248286a85bbdbbc7874165953c1fb3cc5213714227675c5
Sep 16 13:15:00.450070 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:00.450041    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-8d2eac-29826075-89wgf"]
Sep 16 13:15:00.451637 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:15:00.451612    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb590e8f3_19b0_4c1f_8028_b081f89436e4.slice/crio-4cb78494ddba46646f080e3c93d26ec2b5cdb8df430e876dff9f0f9d9ee13fed WatchSource:0}: Error finding container 4cb78494ddba46646f080e3c93d26ec2b5cdb8df430e876dff9f0f9d9ee13fed: Status 404 returned error can't find the container with id 4cb78494ddba46646f080e3c93d26ec2b5cdb8df430e876dff9f0f9d9ee13fed
Sep 16 13:15:00.459918 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:00.459895    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-8d2eac-29826075-89wgf" event={"ID":"b590e8f3-19b0-4c1f-8028-b081f89436e4","Type":"ContainerStarted","Data":"4cb78494ddba46646f080e3c93d26ec2b5cdb8df430e876dff9f0f9d9ee13fed"}
Sep 16 13:15:00.461292 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:00.461272    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a17de2-lmgzx" event={"ID":"557d388b-b04b-4d34-a6a6-7aec9824fd3f","Type":"ContainerStarted","Data":"312836c7b31acab60248286a85bbdbbc7874165953c1fb3cc5213714227675c5"}
Sep 16 13:15:00.468263 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:00.468168    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e344f9-29826075-ftnsd"]
Sep 16 13:15:00.471829 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:15:00.471757    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7db61f0f_b90d_4171_8b6b_d30d542ca9a3.slice/crio-856c3555441f17443ee1370f67cbc799f7fc8605876e7ec731a57327752d5bf0 WatchSource:0}: Error finding container 856c3555441f17443ee1370f67cbc799f7fc8605876e7ec731a57327752d5bf0: Status 404 returned error can't find the container with id 856c3555441f17443ee1370f67cbc799f7fc8605876e7ec731a57327752d5bf0
Sep 16 13:15:00.500530 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:00.500506    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826075-mtpck"]
Sep 16 13:15:00.502445 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:15:00.502423    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6396d36_b7ed_4c46_9ce9_c6cc3608e570.slice/crio-505305b96d4fb5b898a89dacd034b5d2bbc4626eddc70740edb4d7a0adc94f7c WatchSource:0}: Error finding container 505305b96d4fb5b898a89dacd034b5d2bbc4626eddc70740edb4d7a0adc94f7c: Status 404 returned error can't find the container with id 505305b96d4fb5b898a89dacd034b5d2bbc4626eddc70740edb4d7a0adc94f7c
Sep 16 13:15:01.468580 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:01.468529    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826075-mtpck" event={"ID":"a6396d36-b7ed-4c46-9ce9-c6cc3608e570","Type":"ContainerStarted","Data":"e3c65f40ddc60fd1177924564e2e72ad100b7265837f42b46efa095b1ebe1769"}
Sep 16 13:15:01.468580 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:01.468586    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826075-mtpck" event={"ID":"a6396d36-b7ed-4c46-9ce9-c6cc3608e570","Type":"ContainerStarted","Data":"505305b96d4fb5b898a89dacd034b5d2bbc4626eddc70740edb4d7a0adc94f7c"}
Sep 16 13:15:01.470356 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:01.470327    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e344f9-29826075-ftnsd" event={"ID":"7db61f0f-b90d-4171-8b6b-d30d542ca9a3","Type":"ContainerStarted","Data":"d581be2322b6e0a620691ff7c137071df46f69eea7e29564f2dcaf434b5fdafd"}
Sep 16 13:15:01.470479 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:01.470378    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e344f9-29826075-ftnsd" event={"ID":"7db61f0f-b90d-4171-8b6b-d30d542ca9a3","Type":"ContainerStarted","Data":"856c3555441f17443ee1370f67cbc799f7fc8605876e7ec731a57327752d5bf0"}
Sep 16 13:15:01.471892 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:01.471869    2577 generic.go:356] "Generic (PLEG): container finished" podID="557d388b-b04b-4d34-a6a6-7aec9824fd3f" containerID="76d1bc286f1cc3e691338c5a720394047874319cbd634ac3fd72304044d294b8" exitCode=0
Sep 16 13:15:01.472002 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:01.471936    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a17de2-lmgzx" event={"ID":"557d388b-b04b-4d34-a6a6-7aec9824fd3f","Type":"ContainerDied","Data":"76d1bc286f1cc3e691338c5a720394047874319cbd634ac3fd72304044d294b8"}
Sep 16 13:15:01.473460 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:01.473438    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-8d2eac-29826075-89wgf" event={"ID":"b590e8f3-19b0-4c1f-8028-b081f89436e4","Type":"ContainerStarted","Data":"b7d4c126abef7775a200589bd2a36f016fea69eac6b717085bb478edadca6a80"}
Sep 16 13:15:01.485180 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:01.485139    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826075-mtpck" podStartSLOduration=1.485126038 podStartE2EDuration="1.485126038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:15:01.482866212 +0000 UTC m=+2207.930098856" watchObservedRunningTime="2026-09-16 13:15:01.485126038 +0000 UTC m=+2207.932358680"
Sep 16 13:15:01.497887 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:01.497843    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-8d2eac-29826075-89wgf" podStartSLOduration=1.497830535 podStartE2EDuration="1.497830535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:15:01.496366045 +0000 UTC m=+2207.943598688" watchObservedRunningTime="2026-09-16 13:15:01.497830535 +0000 UTC m=+2207.945063177"
Sep 16 13:15:01.511390 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:01.511351    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e344f9-29826075-ftnsd" podStartSLOduration=1.511333748 podStartE2EDuration="1.511333748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:15:01.510200194 +0000 UTC m=+2207.957432836" watchObservedRunningTime="2026-09-16 13:15:01.511333748 +0000 UTC m=+2207.958566395"
Sep 16 13:15:02.481532 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:02.481497    2577 generic.go:356] "Generic (PLEG): container finished" podID="b590e8f3-19b0-4c1f-8028-b081f89436e4" containerID="b7d4c126abef7775a200589bd2a36f016fea69eac6b717085bb478edadca6a80" exitCode=0
Sep 16 13:15:02.481946 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:02.481581    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-8d2eac-29826075-89wgf" event={"ID":"b590e8f3-19b0-4c1f-8028-b081f89436e4","Type":"ContainerDied","Data":"b7d4c126abef7775a200589bd2a36f016fea69eac6b717085bb478edadca6a80"}
Sep 16 13:15:02.483181 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:02.483157    2577 generic.go:356] "Generic (PLEG): container finished" podID="a6396d36-b7ed-4c46-9ce9-c6cc3608e570" containerID="e3c65f40ddc60fd1177924564e2e72ad100b7265837f42b46efa095b1ebe1769" exitCode=0
Sep 16 13:15:02.483274 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:02.483229    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826075-mtpck" event={"ID":"a6396d36-b7ed-4c46-9ce9-c6cc3608e570","Type":"ContainerDied","Data":"e3c65f40ddc60fd1177924564e2e72ad100b7265837f42b46efa095b1ebe1769"}
Sep 16 13:15:02.484725 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:02.484693    2577 generic.go:356] "Generic (PLEG): container finished" podID="7db61f0f-b90d-4171-8b6b-d30d542ca9a3" containerID="d581be2322b6e0a620691ff7c137071df46f69eea7e29564f2dcaf434b5fdafd" exitCode=0
Sep 16 13:15:02.484885 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:02.484772    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e344f9-29826075-ftnsd" event={"ID":"7db61f0f-b90d-4171-8b6b-d30d542ca9a3","Type":"ContainerDied","Data":"d581be2322b6e0a620691ff7c137071df46f69eea7e29564f2dcaf434b5fdafd"}
Sep 16 13:15:02.629440 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:02.629416    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a17de2-lmgzx"
Sep 16 13:15:02.722472 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:02.722442    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/557d388b-b04b-4d34-a6a6-7aec9824fd3f-maas-api-service-ca\") pod \"557d388b-b04b-4d34-a6a6-7aec9824fd3f\" (UID: \"557d388b-b04b-4d34-a6a6-7aec9824fd3f\") "
Sep 16 13:15:02.722787 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:02.722766    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/557d388b-b04b-4d34-a6a6-7aec9824fd3f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "557d388b-b04b-4d34-a6a6-7aec9824fd3f" (UID: "557d388b-b04b-4d34-a6a6-7aec9824fd3f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:15:02.823462 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:02.823429    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/557d388b-b04b-4d34-a6a6-7aec9824fd3f-maas-api-service-ca\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:15:03.492303 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:03.492265    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a17de2-lmgzx" event={"ID":"557d388b-b04b-4d34-a6a6-7aec9824fd3f","Type":"ContainerDied","Data":"312836c7b31acab60248286a85bbdbbc7874165953c1fb3cc5213714227675c5"}
Sep 16 13:15:03.492754 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:03.492327    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="312836c7b31acab60248286a85bbdbbc7874165953c1fb3cc5213714227675c5"
Sep 16 13:15:03.492754 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:03.492509    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a17de2-lmgzx"
Sep 16 13:15:03.644522 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:03.644495    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826075-mtpck"
Sep 16 13:15:03.674048 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:03.674001    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-8d2eac-29826075-89wgf"
Sep 16 13:15:03.716440 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:03.716414    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e344f9-29826075-ftnsd"
Sep 16 13:15:04.498979 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:04.498895    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-8d2eac-29826075-89wgf"
Sep 16 13:15:04.498979 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:04.498905    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-8d2eac-29826075-89wgf" event={"ID":"b590e8f3-19b0-4c1f-8028-b081f89436e4","Type":"ContainerDied","Data":"4cb78494ddba46646f080e3c93d26ec2b5cdb8df430e876dff9f0f9d9ee13fed"}
Sep 16 13:15:04.498979 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:04.498951    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cb78494ddba46646f080e3c93d26ec2b5cdb8df430e876dff9f0f9d9ee13fed"
Sep 16 13:15:04.500476 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:04.500457    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826075-mtpck"
Sep 16 13:15:04.500617 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:04.500455    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826075-mtpck" event={"ID":"a6396d36-b7ed-4c46-9ce9-c6cc3608e570","Type":"ContainerDied","Data":"505305b96d4fb5b898a89dacd034b5d2bbc4626eddc70740edb4d7a0adc94f7c"}
Sep 16 13:15:04.500617 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:04.500589    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="505305b96d4fb5b898a89dacd034b5d2bbc4626eddc70740edb4d7a0adc94f7c"
Sep 16 13:15:04.501956 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:04.501927    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e344f9-29826075-ftnsd"
Sep 16 13:15:04.502071 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:04.501968    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e344f9-29826075-ftnsd" event={"ID":"7db61f0f-b90d-4171-8b6b-d30d542ca9a3","Type":"ContainerDied","Data":"856c3555441f17443ee1370f67cbc799f7fc8605876e7ec731a57327752d5bf0"}
Sep 16 13:15:04.502071 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:04.502015    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="856c3555441f17443ee1370f67cbc799f7fc8605876e7ec731a57327752d5bf0"
Sep 16 13:15:15.144299 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.144267    2577 scope.go:117] "RemoveContainer" containerID="73543931499ac12c8dfd9bbe5a31c29b050296f4c62c1f5f99ca9a4951b91bf5"
Sep 16 13:15:15.153300 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.153271    2577 scope.go:117] "RemoveContainer" containerID="6c2eeca03991e3c1ca642ae8cc0d4a37fd8607dd52cd631407a89d3db2a799b4"
Sep 16 13:15:15.161772 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.161752    2577 scope.go:117] "RemoveContainer" containerID="e285bd0d6fab27393493fe9068afc0fe1e83fe2327eed89b33f5be92366e74b7"
Sep 16 13:15:15.245060 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.245007    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a17de2-669fc94f85-t5cql"]
Sep 16 13:15:15.245338 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.245303    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a17de2-669fc94f85-t5cql" podUID="bf4ed91e-8b4d-41a4-be55-9fa3da6a9302" containerName="payload-processing" containerID="cri-o://ec9d30f46fea38d4e3c5cb7dd3cd1f8ee02c8e85f52786a57cea89de80f9d3e0" gracePeriod=30
Sep 16 13:15:15.252910 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.252881    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a17de2-54494c4c98-q8gtg"]
Sep 16 13:15:15.253171 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.253117    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a17de2-54494c4c98-q8gtg" podUID="20c47bfb-9cfc-4105-8f1b-9b2a505e68ec" containerName="payload-pre-processing" containerID="cri-o://42a8ecb0c4db5ec663a15675f27306ca8573344b113bf21e6e714586f15d048e" gracePeriod=30
Sep 16 13:15:15.408821 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.408717    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a17de2-lmgzx"]
Sep 16 13:15:15.412009 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.411979    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a17de2-lmgzx"]
Sep 16 13:15:15.541081 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.541058    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a17de2-669fc94f85-t5cql"
Sep 16 13:15:15.544158 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.544137    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a17de2-54494c4c98-q8gtg"
Sep 16 13:15:15.569218 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.569069    2577 generic.go:356] "Generic (PLEG): container finished" podID="20c47bfb-9cfc-4105-8f1b-9b2a505e68ec" containerID="42a8ecb0c4db5ec663a15675f27306ca8573344b113bf21e6e714586f15d048e" exitCode=0
Sep 16 13:15:15.569455 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.569224    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a17de2-54494c4c98-q8gtg" event={"ID":"20c47bfb-9cfc-4105-8f1b-9b2a505e68ec","Type":"ContainerDied","Data":"42a8ecb0c4db5ec663a15675f27306ca8573344b113bf21e6e714586f15d048e"}
Sep 16 13:15:15.569455 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.569272    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a17de2-54494c4c98-q8gtg" event={"ID":"20c47bfb-9cfc-4105-8f1b-9b2a505e68ec","Type":"ContainerDied","Data":"358c296a5c4845fe7bbdbd04e0a7ce30f8e33d4b8243c499105138a94e2a9919"}
Sep 16 13:15:15.569455 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.569275    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a17de2-54494c4c98-q8gtg"
Sep 16 13:15:15.569455 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.569305    2577 scope.go:117] "RemoveContainer" containerID="42a8ecb0c4db5ec663a15675f27306ca8573344b113bf21e6e714586f15d048e"
Sep 16 13:15:15.574311 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.572130    2577 generic.go:356] "Generic (PLEG): container finished" podID="bf4ed91e-8b4d-41a4-be55-9fa3da6a9302" containerID="ec9d30f46fea38d4e3c5cb7dd3cd1f8ee02c8e85f52786a57cea89de80f9d3e0" exitCode=0
Sep 16 13:15:15.574311 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.572246    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a17de2-669fc94f85-t5cql" event={"ID":"bf4ed91e-8b4d-41a4-be55-9fa3da6a9302","Type":"ContainerDied","Data":"ec9d30f46fea38d4e3c5cb7dd3cd1f8ee02c8e85f52786a57cea89de80f9d3e0"}
Sep 16 13:15:15.574311 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.572334    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a17de2-669fc94f85-t5cql" event={"ID":"bf4ed91e-8b4d-41a4-be55-9fa3da6a9302","Type":"ContainerDied","Data":"91dff7dd1ea244d1f65bb8009926a822db29178824e1d77368a7389b68e0b54f"}
Sep 16 13:15:15.574311 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.572525    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a17de2-669fc94f85-t5cql"
Sep 16 13:15:15.583396 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.583373    2577 scope.go:117] "RemoveContainer" containerID="42a8ecb0c4db5ec663a15675f27306ca8573344b113bf21e6e714586f15d048e"
Sep 16 13:15:15.584908 ip-10-0-133-190 kubenswrapper[2577]: E0916 13:15:15.583759    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42a8ecb0c4db5ec663a15675f27306ca8573344b113bf21e6e714586f15d048e\": container with ID starting with 42a8ecb0c4db5ec663a15675f27306ca8573344b113bf21e6e714586f15d048e not found: ID does not exist" containerID="42a8ecb0c4db5ec663a15675f27306ca8573344b113bf21e6e714586f15d048e"
Sep 16 13:15:15.584908 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.583794    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42a8ecb0c4db5ec663a15675f27306ca8573344b113bf21e6e714586f15d048e"} err="failed to get container status \"42a8ecb0c4db5ec663a15675f27306ca8573344b113bf21e6e714586f15d048e\": rpc error: code = NotFound desc = could not find container \"42a8ecb0c4db5ec663a15675f27306ca8573344b113bf21e6e714586f15d048e\": container with ID starting with 42a8ecb0c4db5ec663a15675f27306ca8573344b113bf21e6e714586f15d048e not found: ID does not exist"
Sep 16 13:15:15.584908 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.583816    2577 scope.go:117] "RemoveContainer" containerID="ec9d30f46fea38d4e3c5cb7dd3cd1f8ee02c8e85f52786a57cea89de80f9d3e0"
Sep 16 13:15:15.597846 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.597812    2577 scope.go:117] "RemoveContainer" containerID="ec9d30f46fea38d4e3c5cb7dd3cd1f8ee02c8e85f52786a57cea89de80f9d3e0"
Sep 16 13:15:15.598315 ip-10-0-133-190 kubenswrapper[2577]: E0916 13:15:15.598225    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec9d30f46fea38d4e3c5cb7dd3cd1f8ee02c8e85f52786a57cea89de80f9d3e0\": container with ID starting with ec9d30f46fea38d4e3c5cb7dd3cd1f8ee02c8e85f52786a57cea89de80f9d3e0 not found: ID does not exist" containerID="ec9d30f46fea38d4e3c5cb7dd3cd1f8ee02c8e85f52786a57cea89de80f9d3e0"
Sep 16 13:15:15.598315 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.598259    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec9d30f46fea38d4e3c5cb7dd3cd1f8ee02c8e85f52786a57cea89de80f9d3e0"} err="failed to get container status \"ec9d30f46fea38d4e3c5cb7dd3cd1f8ee02c8e85f52786a57cea89de80f9d3e0\": rpc error: code = NotFound desc = could not find container \"ec9d30f46fea38d4e3c5cb7dd3cd1f8ee02c8e85f52786a57cea89de80f9d3e0\": container with ID starting with ec9d30f46fea38d4e3c5cb7dd3cd1f8ee02c8e85f52786a57cea89de80f9d3e0 not found: ID does not exist"
Sep 16 13:15:15.652804 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.652775    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf4ed91e-8b4d-41a4-be55-9fa3da6a9302-plugins-config-volume\") pod \"bf4ed91e-8b4d-41a4-be55-9fa3da6a9302\" (UID: \"bf4ed91e-8b4d-41a4-be55-9fa3da6a9302\") "
Sep 16 13:15:15.652976 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.652873    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c47bfb-9cfc-4105-8f1b-9b2a505e68ec-plugins-config-volume\") pod \"20c47bfb-9cfc-4105-8f1b-9b2a505e68ec\" (UID: \"20c47bfb-9cfc-4105-8f1b-9b2a505e68ec\") "
Sep 16 13:15:15.652976 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.652961    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rtvp\" (UniqueName: \"kubernetes.io/projected/20c47bfb-9cfc-4105-8f1b-9b2a505e68ec-kube-api-access-6rtvp\") pod \"20c47bfb-9cfc-4105-8f1b-9b2a505e68ec\" (UID: \"20c47bfb-9cfc-4105-8f1b-9b2a505e68ec\") "
Sep 16 13:15:15.653121 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.653070    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxw79\" (UniqueName: \"kubernetes.io/projected/bf4ed91e-8b4d-41a4-be55-9fa3da6a9302-kube-api-access-cxw79\") pod \"bf4ed91e-8b4d-41a4-be55-9fa3da6a9302\" (UID: \"bf4ed91e-8b4d-41a4-be55-9fa3da6a9302\") "
Sep 16 13:15:15.653264 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.653243    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf4ed91e-8b4d-41a4-be55-9fa3da6a9302-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bf4ed91e-8b4d-41a4-be55-9fa3da6a9302" (UID: "bf4ed91e-8b4d-41a4-be55-9fa3da6a9302"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:15:15.653342 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.653274    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20c47bfb-9cfc-4105-8f1b-9b2a505e68ec-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "20c47bfb-9cfc-4105-8f1b-9b2a505e68ec" (UID: "20c47bfb-9cfc-4105-8f1b-9b2a505e68ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:15:15.653471 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.653443    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf4ed91e-8b4d-41a4-be55-9fa3da6a9302-plugins-config-volume\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:15:15.653471 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.653459    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c47bfb-9cfc-4105-8f1b-9b2a505e68ec-plugins-config-volume\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:15:15.655279 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.655256    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf4ed91e-8b4d-41a4-be55-9fa3da6a9302-kube-api-access-cxw79" (OuterVolumeSpecName: "kube-api-access-cxw79") pod "bf4ed91e-8b4d-41a4-be55-9fa3da6a9302" (UID: "bf4ed91e-8b4d-41a4-be55-9fa3da6a9302"). InnerVolumeSpecName "kube-api-access-cxw79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:15:15.655357 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.655304    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20c47bfb-9cfc-4105-8f1b-9b2a505e68ec-kube-api-access-6rtvp" (OuterVolumeSpecName: "kube-api-access-6rtvp") pod "20c47bfb-9cfc-4105-8f1b-9b2a505e68ec" (UID: "20c47bfb-9cfc-4105-8f1b-9b2a505e68ec"). InnerVolumeSpecName "kube-api-access-6rtvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:15:15.754308 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.754276    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rtvp\" (UniqueName: \"kubernetes.io/projected/20c47bfb-9cfc-4105-8f1b-9b2a505e68ec-kube-api-access-6rtvp\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:15:15.754308 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.754304    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxw79\" (UniqueName: \"kubernetes.io/projected/bf4ed91e-8b4d-41a4-be55-9fa3da6a9302-kube-api-access-cxw79\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:15:15.891949 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.891923    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a17de2-54494c4c98-q8gtg"]
Sep 16 13:15:15.897508 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.897485    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a17de2-54494c4c98-q8gtg"]
Sep 16 13:15:15.910062 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.910017    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a17de2-669fc94f85-t5cql"]
Sep 16 13:15:15.916690 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:15.916635    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a17de2-669fc94f85-t5cql"]
Sep 16 13:15:16.056476 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:16.056441    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20c47bfb-9cfc-4105-8f1b-9b2a505e68ec" path="/var/lib/kubelet/pods/20c47bfb-9cfc-4105-8f1b-9b2a505e68ec/volumes"
Sep 16 13:15:16.056781 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:16.056768    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="557d388b-b04b-4d34-a6a6-7aec9824fd3f" path="/var/lib/kubelet/pods/557d388b-b04b-4d34-a6a6-7aec9824fd3f/volumes"
Sep 16 13:15:16.057099 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:16.057086    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf4ed91e-8b4d-41a4-be55-9fa3da6a9302" path="/var/lib/kubelet/pods/bf4ed91e-8b4d-41a4-be55-9fa3da6a9302/volumes"
Sep 16 13:15:20.765389 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:20.765349    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a17de2/ipp-route-a17de2-kserve-6c6d888c47-lpx7c"]
Sep 16 13:15:35.785788 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:35.785689    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-a17de2/ipp-route-a17de2-kserve-6c6d888c47-lpx7c" podUID="59b8b489-8e35-456b-8e40-0d08ec849f6f" containerName="main" containerID="cri-o://cfa4f73883646aa6d70bda191c1c73daa015ed8fd85e9ed42051c84d25006730" gracePeriod=45
Sep 16 13:15:36.051958 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:36.051883    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a17de2/ipp-route-a17de2-kserve-6c6d888c47-lpx7c"
Sep 16 13:15:36.143322 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:36.143280    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59b8b489-8e35-456b-8e40-0d08ec849f6f-tls-certs\") pod \"59b8b489-8e35-456b-8e40-0d08ec849f6f\" (UID: \"59b8b489-8e35-456b-8e40-0d08ec849f6f\") "
Sep 16 13:15:36.145624 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:36.145596    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59b8b489-8e35-456b-8e40-0d08ec849f6f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "59b8b489-8e35-456b-8e40-0d08ec849f6f" (UID: "59b8b489-8e35-456b-8e40-0d08ec849f6f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:15:36.244379 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:36.244347    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59b8b489-8e35-456b-8e40-0d08ec849f6f-tmp-dir\") pod \"59b8b489-8e35-456b-8e40-0d08ec849f6f\" (UID: \"59b8b489-8e35-456b-8e40-0d08ec849f6f\") "
Sep 16 13:15:36.244555 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:36.244397    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59b8b489-8e35-456b-8e40-0d08ec849f6f-home\") pod \"59b8b489-8e35-456b-8e40-0d08ec849f6f\" (UID: \"59b8b489-8e35-456b-8e40-0d08ec849f6f\") "
Sep 16 13:15:36.244555 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:36.244420    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59b8b489-8e35-456b-8e40-0d08ec849f6f-model-cache\") pod \"59b8b489-8e35-456b-8e40-0d08ec849f6f\" (UID: \"59b8b489-8e35-456b-8e40-0d08ec849f6f\") "
Sep 16 13:15:36.244555 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:36.244447    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrzcg\" (UniqueName: \"kubernetes.io/projected/59b8b489-8e35-456b-8e40-0d08ec849f6f-kube-api-access-mrzcg\") pod \"59b8b489-8e35-456b-8e40-0d08ec849f6f\" (UID: \"59b8b489-8e35-456b-8e40-0d08ec849f6f\") "
Sep 16 13:15:36.244555 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:36.244464    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59b8b489-8e35-456b-8e40-0d08ec849f6f-dshm\") pod \"59b8b489-8e35-456b-8e40-0d08ec849f6f\" (UID: \"59b8b489-8e35-456b-8e40-0d08ec849f6f\") "
Sep 16 13:15:36.244785 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:36.244601    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b8b489-8e35-456b-8e40-0d08ec849f6f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "59b8b489-8e35-456b-8e40-0d08ec849f6f" (UID: "59b8b489-8e35-456b-8e40-0d08ec849f6f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:15:36.244785 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:36.244680    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b8b489-8e35-456b-8e40-0d08ec849f6f-model-cache" (OuterVolumeSpecName: "model-cache") pod "59b8b489-8e35-456b-8e40-0d08ec849f6f" (UID: "59b8b489-8e35-456b-8e40-0d08ec849f6f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:15:36.244785 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:36.244695    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b8b489-8e35-456b-8e40-0d08ec849f6f-home" (OuterVolumeSpecName: "home") pod "59b8b489-8e35-456b-8e40-0d08ec849f6f" (UID: "59b8b489-8e35-456b-8e40-0d08ec849f6f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:15:36.244922 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:36.244787    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59b8b489-8e35-456b-8e40-0d08ec849f6f-home\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:15:36.244922 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:36.244802    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59b8b489-8e35-456b-8e40-0d08ec849f6f-model-cache\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:15:36.244922 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:36.244815    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59b8b489-8e35-456b-8e40-0d08ec849f6f-tls-certs\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:15:36.244922 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:36.244823    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59b8b489-8e35-456b-8e40-0d08ec849f6f-tmp-dir\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:15:36.246761 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:36.246739    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b8b489-8e35-456b-8e40-0d08ec849f6f-dshm" (OuterVolumeSpecName: "dshm") pod "59b8b489-8e35-456b-8e40-0d08ec849f6f" (UID: "59b8b489-8e35-456b-8e40-0d08ec849f6f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:15:36.246867 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:36.246757    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b8b489-8e35-456b-8e40-0d08ec849f6f-kube-api-access-mrzcg" (OuterVolumeSpecName: "kube-api-access-mrzcg") pod "59b8b489-8e35-456b-8e40-0d08ec849f6f" (UID: "59b8b489-8e35-456b-8e40-0d08ec849f6f"). InnerVolumeSpecName "kube-api-access-mrzcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:15:36.345616 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:36.345538    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrzcg\" (UniqueName: \"kubernetes.io/projected/59b8b489-8e35-456b-8e40-0d08ec849f6f-kube-api-access-mrzcg\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:15:36.345616 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:36.345563    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59b8b489-8e35-456b-8e40-0d08ec849f6f-dshm\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:15:36.696445 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:36.696360    2577 generic.go:356] "Generic (PLEG): container finished" podID="59b8b489-8e35-456b-8e40-0d08ec849f6f" containerID="cfa4f73883646aa6d70bda191c1c73daa015ed8fd85e9ed42051c84d25006730" exitCode=0
Sep 16 13:15:36.696445 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:36.696409    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a17de2/ipp-route-a17de2-kserve-6c6d888c47-lpx7c"
Sep 16 13:15:36.696661 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:36.696449    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a17de2/ipp-route-a17de2-kserve-6c6d888c47-lpx7c" event={"ID":"59b8b489-8e35-456b-8e40-0d08ec849f6f","Type":"ContainerDied","Data":"cfa4f73883646aa6d70bda191c1c73daa015ed8fd85e9ed42051c84d25006730"}
Sep 16 13:15:36.696661 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:36.696496    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a17de2/ipp-route-a17de2-kserve-6c6d888c47-lpx7c" event={"ID":"59b8b489-8e35-456b-8e40-0d08ec849f6f","Type":"ContainerDied","Data":"cf5476678c2abd1f1b69867f30b8c7c78ee1dd772fb9915eb2c3be09b9afcc7a"}
Sep 16 13:15:36.696661 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:36.696514    2577 scope.go:117] "RemoveContainer" containerID="cfa4f73883646aa6d70bda191c1c73daa015ed8fd85e9ed42051c84d25006730"
Sep 16 13:15:36.706340 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:36.706325    2577 scope.go:117] "RemoveContainer" containerID="cfa4f73883646aa6d70bda191c1c73daa015ed8fd85e9ed42051c84d25006730"
Sep 16 13:15:36.706615 ip-10-0-133-190 kubenswrapper[2577]: E0916 13:15:36.706596    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfa4f73883646aa6d70bda191c1c73daa015ed8fd85e9ed42051c84d25006730\": container with ID starting with cfa4f73883646aa6d70bda191c1c73daa015ed8fd85e9ed42051c84d25006730 not found: ID does not exist" containerID="cfa4f73883646aa6d70bda191c1c73daa015ed8fd85e9ed42051c84d25006730"
Sep 16 13:15:36.706684 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:36.706623    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfa4f73883646aa6d70bda191c1c73daa015ed8fd85e9ed42051c84d25006730"} err="failed to get container status \"cfa4f73883646aa6d70bda191c1c73daa015ed8fd85e9ed42051c84d25006730\": rpc error: code = NotFound desc = could not find container \"cfa4f73883646aa6d70bda191c1c73daa015ed8fd85e9ed42051c84d25006730\": container with ID starting with cfa4f73883646aa6d70bda191c1c73daa015ed8fd85e9ed42051c84d25006730 not found: ID does not exist"
Sep 16 13:15:36.721179 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:36.721155    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a17de2/ipp-route-a17de2-kserve-6c6d888c47-lpx7c"]
Sep 16 13:15:36.727928 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:36.725430    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-a17de2/ipp-route-a17de2-kserve-6c6d888c47-lpx7c"]
Sep 16 13:15:38.055669 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:15:38.055630    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59b8b489-8e35-456b-8e40-0d08ec849f6f" path="/var/lib/kubelet/pods/59b8b489-8e35-456b-8e40-0d08ec849f6f/volumes"
Sep 16 13:16:05.434136 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:05.434095    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8d2eac-spjqm"]
Sep 16 13:16:05.434606 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:05.434588    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7db61f0f-b90d-4171-8b6b-d30d542ca9a3" containerName="cleanup"
Sep 16 13:16:05.434655 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:05.434609    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7db61f0f-b90d-4171-8b6b-d30d542ca9a3" containerName="cleanup"
Sep 16 13:16:05.434655 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:05.434622    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20c47bfb-9cfc-4105-8f1b-9b2a505e68ec" containerName="payload-pre-processing"
Sep 16 13:16:05.434655 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:05.434628    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="20c47bfb-9cfc-4105-8f1b-9b2a505e68ec" containerName="payload-pre-processing"
Sep 16 13:16:05.434655 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:05.434641    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6396d36-b7ed-4c46-9ce9-c6cc3608e570" containerName="cleanup"
Sep 16 13:16:05.434655 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:05.434647    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6396d36-b7ed-4c46-9ce9-c6cc3608e570" containerName="cleanup"
Sep 16 13:16:05.434655 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:05.434656    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf4ed91e-8b4d-41a4-be55-9fa3da6a9302" containerName="payload-processing"
Sep 16 13:16:05.434840 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:05.434662    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf4ed91e-8b4d-41a4-be55-9fa3da6a9302" containerName="payload-processing"
Sep 16 13:16:05.434840 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:05.434674    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b590e8f3-19b0-4c1f-8028-b081f89436e4" containerName="cleanup"
Sep 16 13:16:05.434840 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:05.434680    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b590e8f3-19b0-4c1f-8028-b081f89436e4" containerName="cleanup"
Sep 16 13:16:05.434840 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:05.434687    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="557d388b-b04b-4d34-a6a6-7aec9824fd3f" containerName="revoke-keys"
Sep 16 13:16:05.434840 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:05.434692    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="557d388b-b04b-4d34-a6a6-7aec9824fd3f" containerName="revoke-keys"
Sep 16 13:16:05.434840 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:05.434700    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59b8b489-8e35-456b-8e40-0d08ec849f6f" containerName="main"
Sep 16 13:16:05.434840 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:05.434705    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="59b8b489-8e35-456b-8e40-0d08ec849f6f" containerName="main"
Sep 16 13:16:05.434840 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:05.434778    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6396d36-b7ed-4c46-9ce9-c6cc3608e570" containerName="cleanup"
Sep 16 13:16:05.434840 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:05.434784    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf4ed91e-8b4d-41a4-be55-9fa3da6a9302" containerName="payload-processing"
Sep 16 13:16:05.434840 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:05.434790    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7db61f0f-b90d-4171-8b6b-d30d542ca9a3" containerName="cleanup"
Sep 16 13:16:05.434840 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:05.434796    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b590e8f3-19b0-4c1f-8028-b081f89436e4" containerName="cleanup"
Sep 16 13:16:05.434840 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:05.434805    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="557d388b-b04b-4d34-a6a6-7aec9824fd3f" containerName="revoke-keys"
Sep 16 13:16:05.434840 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:05.434812    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="59b8b489-8e35-456b-8e40-0d08ec849f6f" containerName="main"
Sep 16 13:16:05.434840 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:05.434816    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="20c47bfb-9cfc-4105-8f1b-9b2a505e68ec" containerName="payload-pre-processing"
Sep 16 13:16:05.438106 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:05.438089    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8d2eac-spjqm"
Sep 16 13:16:05.441250 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:05.441228    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-64zph\""
Sep 16 13:16:05.441382 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:05.441249    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 13:16:05.446374 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:05.446348    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8d2eac-spjqm"]
Sep 16 13:16:05.511085 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:05.511008    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f563b96-db54-4373-9429-2e95d6b20d96-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8d2eac-spjqm\" (UID: \"1f563b96-db54-4373-9429-2e95d6b20d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8d2eac-spjqm"
Sep 16 13:16:05.611848 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:05.611812    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f563b96-db54-4373-9429-2e95d6b20d96-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8d2eac-spjqm\" (UID: \"1f563b96-db54-4373-9429-2e95d6b20d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8d2eac-spjqm"
Sep 16 13:16:05.612460 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:05.612440    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f563b96-db54-4373-9429-2e95d6b20d96-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8d2eac-spjqm\" (UID: \"1f563b96-db54-4373-9429-2e95d6b20d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8d2eac-spjqm"
Sep 16 13:16:05.750994 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:05.750896    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8d2eac-spjqm"
Sep 16 13:16:06.032190 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:06.032167    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8d2eac-spjqm"]
Sep 16 13:16:06.034395 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:16:06.034369    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f563b96_db54_4373_9429_2e95d6b20d96.slice/crio-5f9d47702b1e199a1b09a593c1743526d6c31e575782a23ca7841a2b81f502be WatchSource:0}: Error finding container 5f9d47702b1e199a1b09a593c1743526d6c31e575782a23ca7841a2b81f502be: Status 404 returned error can't find the container with id 5f9d47702b1e199a1b09a593c1743526d6c31e575782a23ca7841a2b81f502be
Sep 16 13:16:06.867018 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:06.866980    2577 generic.go:356] "Generic (PLEG): container finished" podID="1f563b96-db54-4373-9429-2e95d6b20d96" containerID="515f701a94b5630b922e2e9f73d9d5476e2722d98d8f37885d5de8d6d7db81df" exitCode=0
Sep 16 13:16:06.867416 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:06.867074    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8d2eac-spjqm" event={"ID":"1f563b96-db54-4373-9429-2e95d6b20d96","Type":"ContainerDied","Data":"515f701a94b5630b922e2e9f73d9d5476e2722d98d8f37885d5de8d6d7db81df"}
Sep 16 13:16:06.867416 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:06.867124    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8d2eac-spjqm" event={"ID":"1f563b96-db54-4373-9429-2e95d6b20d96","Type":"ContainerStarted","Data":"5f9d47702b1e199a1b09a593c1743526d6c31e575782a23ca7841a2b81f502be"}
Sep 16 13:16:08.010279 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:08.010255    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8d2eac-spjqm"
Sep 16 13:16:08.135016 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:08.134931    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f563b96-db54-4373-9429-2e95d6b20d96-maas-api-service-ca\") pod \"1f563b96-db54-4373-9429-2e95d6b20d96\" (UID: \"1f563b96-db54-4373-9429-2e95d6b20d96\") "
Sep 16 13:16:08.135345 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:08.135316    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f563b96-db54-4373-9429-2e95d6b20d96-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1f563b96-db54-4373-9429-2e95d6b20d96" (UID: "1f563b96-db54-4373-9429-2e95d6b20d96"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:16:08.236387 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:08.236347    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f563b96-db54-4373-9429-2e95d6b20d96-maas-api-service-ca\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:16:08.879438 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:08.879400    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8d2eac-spjqm" event={"ID":"1f563b96-db54-4373-9429-2e95d6b20d96","Type":"ContainerDied","Data":"5f9d47702b1e199a1b09a593c1743526d6c31e575782a23ca7841a2b81f502be"}
Sep 16 13:16:08.879645 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:08.879455    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f9d47702b1e199a1b09a593c1743526d6c31e575782a23ca7841a2b81f502be"
Sep 16 13:16:08.879645 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:08.879412    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8d2eac-spjqm"
Sep 16 13:16:15.204908 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:15.204877    2577 scope.go:117] "RemoveContainer" containerID="3a1cec02ecf5eefc0dd5ab196e1aef23033ac3adce2125a65bc694f77e48e4ec"
Sep 16 13:16:15.216585 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:15.216562    2577 scope.go:117] "RemoveContainer" containerID="1364073e0b10fa0908c6bc1c1b4b3390d567aa54f826e1df2029e4ee4ac6bfa3"
Sep 16 13:16:16.032867 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:16.032830    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-8d2eac-29826075-89wgf"]
Sep 16 13:16:16.035900 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:16.035872    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-8d2eac-29826075-89wgf"]
Sep 16 13:16:16.055581 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:16.055551    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b590e8f3-19b0-4c1f-8028-b081f89436e4" path="/var/lib/kubelet/pods/b590e8f3-19b0-4c1f-8028-b081f89436e4/volumes"
Sep 16 13:16:16.080074 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:16.080044    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8d2eac-6988b44654-frd8b"]
Sep 16 13:16:16.080347 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:16.080296    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d2eac-6988b44654-frd8b" podUID="0b2268c5-3eda-4392-8786-d19a808b3182" containerName="payload-pre-processing" containerID="cri-o://ede9c20e9f6ff35ee00f03d8d9fbda80a3219d8ceaa38b1797c23caca60a8eb2" gracePeriod=30
Sep 16 13:16:16.267893 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:16.267859    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8d2eac-spjqm"]
Sep 16 13:16:16.270704 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:16.270679    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8d2eac-spjqm"]
Sep 16 13:16:16.354777 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:16.354742    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d2eac-6988b44654-frd8b"
Sep 16 13:16:16.517717 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:16.517687    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47vrc\" (UniqueName: \"kubernetes.io/projected/0b2268c5-3eda-4392-8786-d19a808b3182-kube-api-access-47vrc\") pod \"0b2268c5-3eda-4392-8786-d19a808b3182\" (UID: \"0b2268c5-3eda-4392-8786-d19a808b3182\") "
Sep 16 13:16:16.517882 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:16.517748    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b2268c5-3eda-4392-8786-d19a808b3182-plugins-config-volume\") pod \"0b2268c5-3eda-4392-8786-d19a808b3182\" (UID: \"0b2268c5-3eda-4392-8786-d19a808b3182\") "
Sep 16 13:16:16.518117 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:16.518093    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b2268c5-3eda-4392-8786-d19a808b3182-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0b2268c5-3eda-4392-8786-d19a808b3182" (UID: "0b2268c5-3eda-4392-8786-d19a808b3182"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:16:16.518186 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:16.518173    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b2268c5-3eda-4392-8786-d19a808b3182-plugins-config-volume\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:16:16.520054 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:16.519993    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b2268c5-3eda-4392-8786-d19a808b3182-kube-api-access-47vrc" (OuterVolumeSpecName: "kube-api-access-47vrc") pod "0b2268c5-3eda-4392-8786-d19a808b3182" (UID: "0b2268c5-3eda-4392-8786-d19a808b3182"). InnerVolumeSpecName "kube-api-access-47vrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:16:16.619327 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:16.619243    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47vrc\" (UniqueName: \"kubernetes.io/projected/0b2268c5-3eda-4392-8786-d19a808b3182-kube-api-access-47vrc\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:16:16.924782 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:16.924690    2577 generic.go:356] "Generic (PLEG): container finished" podID="0b2268c5-3eda-4392-8786-d19a808b3182" containerID="ede9c20e9f6ff35ee00f03d8d9fbda80a3219d8ceaa38b1797c23caca60a8eb2" exitCode=0
Sep 16 13:16:16.924782 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:16.924740    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d2eac-6988b44654-frd8b" event={"ID":"0b2268c5-3eda-4392-8786-d19a808b3182","Type":"ContainerDied","Data":"ede9c20e9f6ff35ee00f03d8d9fbda80a3219d8ceaa38b1797c23caca60a8eb2"}
Sep 16 13:16:16.924782 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:16.924743    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d2eac-6988b44654-frd8b"
Sep 16 13:16:16.925084 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:16.924807    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d2eac-6988b44654-frd8b" event={"ID":"0b2268c5-3eda-4392-8786-d19a808b3182","Type":"ContainerDied","Data":"85a9527ee287118e95e119b8920b1afd4511d86a343fb76f8e9d39d793cfc462"}
Sep 16 13:16:16.925084 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:16.924829    2577 scope.go:117] "RemoveContainer" containerID="ede9c20e9f6ff35ee00f03d8d9fbda80a3219d8ceaa38b1797c23caca60a8eb2"
Sep 16 13:16:16.934333 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:16.934314    2577 scope.go:117] "RemoveContainer" containerID="ede9c20e9f6ff35ee00f03d8d9fbda80a3219d8ceaa38b1797c23caca60a8eb2"
Sep 16 13:16:16.934618 ip-10-0-133-190 kubenswrapper[2577]: E0916 13:16:16.934599    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ede9c20e9f6ff35ee00f03d8d9fbda80a3219d8ceaa38b1797c23caca60a8eb2\": container with ID starting with ede9c20e9f6ff35ee00f03d8d9fbda80a3219d8ceaa38b1797c23caca60a8eb2 not found: ID does not exist" containerID="ede9c20e9f6ff35ee00f03d8d9fbda80a3219d8ceaa38b1797c23caca60a8eb2"
Sep 16 13:16:16.934700 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:16.934626    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ede9c20e9f6ff35ee00f03d8d9fbda80a3219d8ceaa38b1797c23caca60a8eb2"} err="failed to get container status \"ede9c20e9f6ff35ee00f03d8d9fbda80a3219d8ceaa38b1797c23caca60a8eb2\": rpc error: code = NotFound desc = could not find container \"ede9c20e9f6ff35ee00f03d8d9fbda80a3219d8ceaa38b1797c23caca60a8eb2\": container with ID starting with ede9c20e9f6ff35ee00f03d8d9fbda80a3219d8ceaa38b1797c23caca60a8eb2 not found: ID does not exist"
Sep 16 13:16:16.948039 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:16.947999    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8d2eac-6988b44654-frd8b"]
Sep 16 13:16:16.953246 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:16.953221    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8d2eac-6988b44654-frd8b"]
Sep 16 13:16:18.055269 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:18.055235    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b2268c5-3eda-4392-8786-d19a808b3182" path="/var/lib/kubelet/pods/0b2268c5-3eda-4392-8786-d19a808b3182/volumes"
Sep 16 13:16:18.055651 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:18.055593    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f563b96-db54-4373-9429-2e95d6b20d96" path="/var/lib/kubelet/pods/1f563b96-db54-4373-9429-2e95d6b20d96/volumes"
Sep 16 13:16:23.615256 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:23.615219    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"]
Sep 16 13:16:23.615646 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:23.615492    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc" podUID="47140d4f-0539-4afb-a426-d9df88f312e5" containerName="istio-proxy" containerID="cri-o://f29bc91212e2a68fa2f8cc62acd122a04c1de131ffcb3e918cfe6030d4b944b2" gracePeriod=30
Sep 16 13:16:23.972333 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:23.972250    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1ef95c-9n6b8"]
Sep 16 13:16:23.972698 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:23.972685    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f563b96-db54-4373-9429-2e95d6b20d96" containerName="revoke-keys"
Sep 16 13:16:23.972761 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:23.972700    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f563b96-db54-4373-9429-2e95d6b20d96" containerName="revoke-keys"
Sep 16 13:16:23.972761 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:23.972725    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b2268c5-3eda-4392-8786-d19a808b3182" containerName="payload-pre-processing"
Sep 16 13:16:23.972761 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:23.972730    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b2268c5-3eda-4392-8786-d19a808b3182" containerName="payload-pre-processing"
Sep 16 13:16:23.972868 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:23.972788    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f563b96-db54-4373-9429-2e95d6b20d96" containerName="revoke-keys"
Sep 16 13:16:23.972868 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:23.972798    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b2268c5-3eda-4392-8786-d19a808b3182" containerName="payload-pre-processing"
Sep 16 13:16:23.976066 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:23.976022    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1ef95c-9n6b8"
Sep 16 13:16:23.978632 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:23.978614    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 13:16:23.978730 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:23.978716    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-64zph\""
Sep 16 13:16:23.983527 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:23.983505    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1ef95c-9n6b8"]
Sep 16 13:16:24.089129 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:24.089098    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02b22263-157b-4e9a-bba0-92f52aec0fa5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-1ef95c-9n6b8\" (UID: \"02b22263-157b-4e9a-bba0-92f52aec0fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1ef95c-9n6b8"
Sep 16 13:16:24.189945 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:24.189905    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02b22263-157b-4e9a-bba0-92f52aec0fa5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-1ef95c-9n6b8\" (UID: \"02b22263-157b-4e9a-bba0-92f52aec0fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1ef95c-9n6b8"
Sep 16 13:16:24.190571 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:24.190551    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02b22263-157b-4e9a-bba0-92f52aec0fa5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-1ef95c-9n6b8\" (UID: \"02b22263-157b-4e9a-bba0-92f52aec0fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1ef95c-9n6b8"
Sep 16 13:16:24.288719 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:24.288674    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1ef95c-9n6b8"
Sep 16 13:16:24.565347 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:24.565316    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1ef95c-9n6b8"]
Sep 16 13:16:24.567221 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:16:24.567196    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02b22263_157b_4e9a_bba0_92f52aec0fa5.slice/crio-07d174d84e68f0fc04e426c4a1dc359abf9f137d4ac5507dfc4683732b7c69a3 WatchSource:0}: Error finding container 07d174d84e68f0fc04e426c4a1dc359abf9f137d4ac5507dfc4683732b7c69a3: Status 404 returned error can't find the container with id 07d174d84e68f0fc04e426c4a1dc359abf9f137d4ac5507dfc4683732b7c69a3
Sep 16 13:16:24.972156 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:24.972100    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1ef95c-9n6b8" event={"ID":"02b22263-157b-4e9a-bba0-92f52aec0fa5","Type":"ContainerStarted","Data":"854c3faf6a990497690575ff98fe21baa27aa1bdab8c0b1232998192ac4ec4c1"}
Sep 16 13:16:24.972658 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:24.972188    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1ef95c-9n6b8" event={"ID":"02b22263-157b-4e9a-bba0-92f52aec0fa5","Type":"ContainerStarted","Data":"07d174d84e68f0fc04e426c4a1dc359abf9f137d4ac5507dfc4683732b7c69a3"}
Sep 16 13:16:24.989543 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:24.989476    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1ef95c-9n6b8" podStartSLOduration=1.989454592 podStartE2EDuration="1.989454592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:16:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:16:24.987939545 +0000 UTC m=+2291.435172199" watchObservedRunningTime="2026-09-16 13:16:24.989454592 +0000 UTC m=+2291.436687234"
Sep 16 13:16:25.979496 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:25.979458    2577 generic.go:356] "Generic (PLEG): container finished" podID="02b22263-157b-4e9a-bba0-92f52aec0fa5" containerID="854c3faf6a990497690575ff98fe21baa27aa1bdab8c0b1232998192ac4ec4c1" exitCode=0
Sep 16 13:16:25.979876 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:25.979547    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1ef95c-9n6b8" event={"ID":"02b22263-157b-4e9a-bba0-92f52aec0fa5","Type":"ContainerDied","Data":"854c3faf6a990497690575ff98fe21baa27aa1bdab8c0b1232998192ac4ec4c1"}
Sep 16 13:16:27.123727 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:27.123700    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1ef95c-9n6b8"
Sep 16 13:16:27.223338 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:27.223306    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02b22263-157b-4e9a-bba0-92f52aec0fa5-maas-api-service-ca\") pod \"02b22263-157b-4e9a-bba0-92f52aec0fa5\" (UID: \"02b22263-157b-4e9a-bba0-92f52aec0fa5\") "
Sep 16 13:16:27.223645 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:27.223623    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02b22263-157b-4e9a-bba0-92f52aec0fa5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "02b22263-157b-4e9a-bba0-92f52aec0fa5" (UID: "02b22263-157b-4e9a-bba0-92f52aec0fa5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:16:27.325083 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:27.325043    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02b22263-157b-4e9a-bba0-92f52aec0fa5-maas-api-service-ca\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:16:27.992695 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:27.992652    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1ef95c-9n6b8" event={"ID":"02b22263-157b-4e9a-bba0-92f52aec0fa5","Type":"ContainerDied","Data":"07d174d84e68f0fc04e426c4a1dc359abf9f137d4ac5507dfc4683732b7c69a3"}
Sep 16 13:16:27.992695 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:27.992683    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1ef95c-9n6b8"
Sep 16 13:16:27.992916 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:27.992716    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07d174d84e68f0fc04e426c4a1dc359abf9f137d4ac5507dfc4683732b7c69a3"
Sep 16 13:16:28.859465 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:28.859436    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"
Sep 16 13:16:28.942258 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:28.942145    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47140d4f-0539-4afb-a426-d9df88f312e5-workload-socket\") pod \"47140d4f-0539-4afb-a426-d9df88f312e5\" (UID: \"47140d4f-0539-4afb-a426-d9df88f312e5\") "
Sep 16 13:16:28.942434 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:28.942260    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47140d4f-0539-4afb-a426-d9df88f312e5-istio-podinfo\") pod \"47140d4f-0539-4afb-a426-d9df88f312e5\" (UID: \"47140d4f-0539-4afb-a426-d9df88f312e5\") "
Sep 16 13:16:28.943197 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:28.942295    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47140d4f-0539-4afb-a426-d9df88f312e5-istio-data\") pod \"47140d4f-0539-4afb-a426-d9df88f312e5\" (UID: \"47140d4f-0539-4afb-a426-d9df88f312e5\") "
Sep 16 13:16:28.943197 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:28.942692    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47140d4f-0539-4afb-a426-d9df88f312e5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "47140d4f-0539-4afb-a426-d9df88f312e5" (UID: "47140d4f-0539-4afb-a426-d9df88f312e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:16:28.943197 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:28.942761    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qs2jx\" (UniqueName: \"kubernetes.io/projected/47140d4f-0539-4afb-a426-d9df88f312e5-kube-api-access-qs2jx\") pod \"47140d4f-0539-4afb-a426-d9df88f312e5\" (UID: \"47140d4f-0539-4afb-a426-d9df88f312e5\") "
Sep 16 13:16:28.943197 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:28.942840    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47140d4f-0539-4afb-a426-d9df88f312e5-istio-token\") pod \"47140d4f-0539-4afb-a426-d9df88f312e5\" (UID: \"47140d4f-0539-4afb-a426-d9df88f312e5\") "
Sep 16 13:16:28.943197 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:28.942885    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47140d4f-0539-4afb-a426-d9df88f312e5-istiod-ca-cert\") pod \"47140d4f-0539-4afb-a426-d9df88f312e5\" (UID: \"47140d4f-0539-4afb-a426-d9df88f312e5\") "
Sep 16 13:16:28.943197 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:28.942966    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47140d4f-0539-4afb-a426-d9df88f312e5-credential-socket\") pod \"47140d4f-0539-4afb-a426-d9df88f312e5\" (UID: \"47140d4f-0539-4afb-a426-d9df88f312e5\") "
Sep 16 13:16:28.943197 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:28.942993    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47140d4f-0539-4afb-a426-d9df88f312e5-istio-data" (OuterVolumeSpecName: "istio-data") pod "47140d4f-0539-4afb-a426-d9df88f312e5" (UID: "47140d4f-0539-4afb-a426-d9df88f312e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:16:28.943197 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:28.943098    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47140d4f-0539-4afb-a426-d9df88f312e5-istio-envoy\") pod \"47140d4f-0539-4afb-a426-d9df88f312e5\" (UID: \"47140d4f-0539-4afb-a426-d9df88f312e5\") "
Sep 16 13:16:28.943197 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:28.943191    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47140d4f-0539-4afb-a426-d9df88f312e5-workload-certs\") pod \"47140d4f-0539-4afb-a426-d9df88f312e5\" (UID: \"47140d4f-0539-4afb-a426-d9df88f312e5\") "
Sep 16 13:16:28.944216 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:28.943989    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47140d4f-0539-4afb-a426-d9df88f312e5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "47140d4f-0539-4afb-a426-d9df88f312e5" (UID: "47140d4f-0539-4afb-a426-d9df88f312e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:16:28.944216 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:28.944071    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47140d4f-0539-4afb-a426-d9df88f312e5-workload-socket\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:16:28.944216 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:28.944098    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47140d4f-0539-4afb-a426-d9df88f312e5-istio-data\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:16:28.944824 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:28.944581    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47140d4f-0539-4afb-a426-d9df88f312e5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "47140d4f-0539-4afb-a426-d9df88f312e5" (UID: "47140d4f-0539-4afb-a426-d9df88f312e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:16:28.945227 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:28.944956    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47140d4f-0539-4afb-a426-d9df88f312e5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "47140d4f-0539-4afb-a426-d9df88f312e5" (UID: "47140d4f-0539-4afb-a426-d9df88f312e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:16:28.945762 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:28.945729    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47140d4f-0539-4afb-a426-d9df88f312e5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "47140d4f-0539-4afb-a426-d9df88f312e5" (UID: "47140d4f-0539-4afb-a426-d9df88f312e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 13:16:28.951652 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:28.946286    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47140d4f-0539-4afb-a426-d9df88f312e5-kube-api-access-qs2jx" (OuterVolumeSpecName: "kube-api-access-qs2jx") pod "47140d4f-0539-4afb-a426-d9df88f312e5" (UID: "47140d4f-0539-4afb-a426-d9df88f312e5"). InnerVolumeSpecName "kube-api-access-qs2jx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:16:28.951652 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:28.946550    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47140d4f-0539-4afb-a426-d9df88f312e5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "47140d4f-0539-4afb-a426-d9df88f312e5" (UID: "47140d4f-0539-4afb-a426-d9df88f312e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:16:28.952313 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:28.952288    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47140d4f-0539-4afb-a426-d9df88f312e5-istio-token" (OuterVolumeSpecName: "istio-token") pod "47140d4f-0539-4afb-a426-d9df88f312e5" (UID: "47140d4f-0539-4afb-a426-d9df88f312e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:16:29.000376 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:29.000336    2577 generic.go:356] "Generic (PLEG): container finished" podID="47140d4f-0539-4afb-a426-d9df88f312e5" containerID="f29bc91212e2a68fa2f8cc62acd122a04c1de131ffcb3e918cfe6030d4b944b2" exitCode=0
Sep 16 13:16:29.000532 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:29.000407    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"
Sep 16 13:16:29.000532 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:29.000425    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc" event={"ID":"47140d4f-0539-4afb-a426-d9df88f312e5","Type":"ContainerDied","Data":"f29bc91212e2a68fa2f8cc62acd122a04c1de131ffcb3e918cfe6030d4b944b2"}
Sep 16 13:16:29.000532 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:29.000510    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc" event={"ID":"47140d4f-0539-4afb-a426-d9df88f312e5","Type":"ContainerDied","Data":"134933c86eaea6908b9e7dcd8543faea5a2fbd20b10d1ebe354d94c756dcb929"}
Sep 16 13:16:29.000679 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:29.000532    2577 scope.go:117] "RemoveContainer" containerID="f29bc91212e2a68fa2f8cc62acd122a04c1de131ffcb3e918cfe6030d4b944b2"
Sep 16 13:16:29.011641 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:29.011625    2577 scope.go:117] "RemoveContainer" containerID="1b946ff8ac7a67bd338f5c7b76a4d29518221bde6e65e693d1037a7e7a5259ce"
Sep 16 13:16:29.020851 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:29.020833    2577 scope.go:117] "RemoveContainer" containerID="f29bc91212e2a68fa2f8cc62acd122a04c1de131ffcb3e918cfe6030d4b944b2"
Sep 16 13:16:29.021167 ip-10-0-133-190 kubenswrapper[2577]: E0916 13:16:29.021145    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f29bc91212e2a68fa2f8cc62acd122a04c1de131ffcb3e918cfe6030d4b944b2\": container with ID starting with f29bc91212e2a68fa2f8cc62acd122a04c1de131ffcb3e918cfe6030d4b944b2 not found: ID does not exist" containerID="f29bc91212e2a68fa2f8cc62acd122a04c1de131ffcb3e918cfe6030d4b944b2"
Sep 16 13:16:29.021224 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:29.021177    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f29bc91212e2a68fa2f8cc62acd122a04c1de131ffcb3e918cfe6030d4b944b2"} err="failed to get container status \"f29bc91212e2a68fa2f8cc62acd122a04c1de131ffcb3e918cfe6030d4b944b2\": rpc error: code = NotFound desc = could not find container \"f29bc91212e2a68fa2f8cc62acd122a04c1de131ffcb3e918cfe6030d4b944b2\": container with ID starting with f29bc91212e2a68fa2f8cc62acd122a04c1de131ffcb3e918cfe6030d4b944b2 not found: ID does not exist"
Sep 16 13:16:29.021224 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:29.021196    2577 scope.go:117] "RemoveContainer" containerID="1b946ff8ac7a67bd338f5c7b76a4d29518221bde6e65e693d1037a7e7a5259ce"
Sep 16 13:16:29.021517 ip-10-0-133-190 kubenswrapper[2577]: E0916 13:16:29.021499    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b946ff8ac7a67bd338f5c7b76a4d29518221bde6e65e693d1037a7e7a5259ce\": container with ID starting with 1b946ff8ac7a67bd338f5c7b76a4d29518221bde6e65e693d1037a7e7a5259ce not found: ID does not exist" containerID="1b946ff8ac7a67bd338f5c7b76a4d29518221bde6e65e693d1037a7e7a5259ce"
Sep 16 13:16:29.021564 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:29.021523    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b946ff8ac7a67bd338f5c7b76a4d29518221bde6e65e693d1037a7e7a5259ce"} err="failed to get container status \"1b946ff8ac7a67bd338f5c7b76a4d29518221bde6e65e693d1037a7e7a5259ce\": rpc error: code = NotFound desc = could not find container \"1b946ff8ac7a67bd338f5c7b76a4d29518221bde6e65e693d1037a7e7a5259ce\": container with ID starting with 1b946ff8ac7a67bd338f5c7b76a4d29518221bde6e65e693d1037a7e7a5259ce not found: ID does not exist"
Sep 16 13:16:29.025149 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:29.025123    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"]
Sep 16 13:16:29.032444 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:29.032420    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8d2eac-openshift-default-5f64964966-wvsqc"]
Sep 16 13:16:29.044869 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:29.044847    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47140d4f-0539-4afb-a426-d9df88f312e5-credential-socket\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:16:29.044869 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:29.044868    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47140d4f-0539-4afb-a426-d9df88f312e5-istio-envoy\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:16:29.045005 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:29.044879    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47140d4f-0539-4afb-a426-d9df88f312e5-workload-certs\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:16:29.045005 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:29.044886    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47140d4f-0539-4afb-a426-d9df88f312e5-istio-podinfo\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:16:29.045005 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:29.044895    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs2jx\" (UniqueName: \"kubernetes.io/projected/47140d4f-0539-4afb-a426-d9df88f312e5-kube-api-access-qs2jx\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:16:29.045005 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:29.044903    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47140d4f-0539-4afb-a426-d9df88f312e5-istio-token\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:16:29.045005 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:29.044910    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47140d4f-0539-4afb-a426-d9df88f312e5-istiod-ca-cert\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:16:30.055402 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:30.055365    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47140d4f-0539-4afb-a426-d9df88f312e5" path="/var/lib/kubelet/pods/47140d4f-0539-4afb-a426-d9df88f312e5/volumes"
Sep 16 13:16:34.825044 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:34.825004    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1ef95c-9dd877f6f-8nkkc"]
Sep 16 13:16:34.825481 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:34.825228    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ef95c-9dd877f6f-8nkkc" podUID="0971a37b-ff6f-446c-aafb-df8e049af2db" containerName="payload-processing" containerID="cri-o://90969b3b92ea3983cf5b9358deedf0f7345d6a0206762764b6cff695d2726b51" gracePeriod=30
Sep 16 13:16:35.022740 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:35.022670    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1ef95c-9n6b8"]
Sep 16 13:16:35.025751 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:35.025726    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1ef95c-9n6b8"]
Sep 16 13:16:35.037581 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:35.037546    2577 generic.go:356] "Generic (PLEG): container finished" podID="0971a37b-ff6f-446c-aafb-df8e049af2db" containerID="90969b3b92ea3983cf5b9358deedf0f7345d6a0206762764b6cff695d2726b51" exitCode=0
Sep 16 13:16:35.037714 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:35.037622    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ef95c-9dd877f6f-8nkkc" event={"ID":"0971a37b-ff6f-446c-aafb-df8e049af2db","Type":"ContainerDied","Data":"90969b3b92ea3983cf5b9358deedf0f7345d6a0206762764b6cff695d2726b51"}
Sep 16 13:16:35.095153 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:35.095118    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ef95c-9dd877f6f-8nkkc"
Sep 16 13:16:35.199878 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:35.199838    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88z2v\" (UniqueName: \"kubernetes.io/projected/0971a37b-ff6f-446c-aafb-df8e049af2db-kube-api-access-88z2v\") pod \"0971a37b-ff6f-446c-aafb-df8e049af2db\" (UID: \"0971a37b-ff6f-446c-aafb-df8e049af2db\") "
Sep 16 13:16:35.200082 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:35.199934    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0971a37b-ff6f-446c-aafb-df8e049af2db-plugins-config-volume\") pod \"0971a37b-ff6f-446c-aafb-df8e049af2db\" (UID: \"0971a37b-ff6f-446c-aafb-df8e049af2db\") "
Sep 16 13:16:35.200370 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:35.200344    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0971a37b-ff6f-446c-aafb-df8e049af2db-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0971a37b-ff6f-446c-aafb-df8e049af2db" (UID: "0971a37b-ff6f-446c-aafb-df8e049af2db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:16:35.202099 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:35.202079    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0971a37b-ff6f-446c-aafb-df8e049af2db-kube-api-access-88z2v" (OuterVolumeSpecName: "kube-api-access-88z2v") pod "0971a37b-ff6f-446c-aafb-df8e049af2db" (UID: "0971a37b-ff6f-446c-aafb-df8e049af2db"). InnerVolumeSpecName "kube-api-access-88z2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:16:35.301208 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:35.301178    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0971a37b-ff6f-446c-aafb-df8e049af2db-plugins-config-volume\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:16:35.301208 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:35.301203    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88z2v\" (UniqueName: \"kubernetes.io/projected/0971a37b-ff6f-446c-aafb-df8e049af2db-kube-api-access-88z2v\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:16:36.044188 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:36.044154    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ef95c-9dd877f6f-8nkkc"
Sep 16 13:16:36.044583 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:36.044191    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ef95c-9dd877f6f-8nkkc" event={"ID":"0971a37b-ff6f-446c-aafb-df8e049af2db","Type":"ContainerDied","Data":"605b04b4f646abd01aeb20c31616abf07e56ac4d03b9a93d26ba823bc46784f6"}
Sep 16 13:16:36.044583 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:36.044260    2577 scope.go:117] "RemoveContainer" containerID="90969b3b92ea3983cf5b9358deedf0f7345d6a0206762764b6cff695d2726b51"
Sep 16 13:16:36.055172 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:36.055150    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02b22263-157b-4e9a-bba0-92f52aec0fa5" path="/var/lib/kubelet/pods/02b22263-157b-4e9a-bba0-92f52aec0fa5/volumes"
Sep 16 13:16:36.067508 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:36.067484    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1ef95c-9dd877f6f-8nkkc"]
Sep 16 13:16:36.073440 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:36.073417    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1ef95c-9dd877f6f-8nkkc"]
Sep 16 13:16:38.055058 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:38.055003    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0971a37b-ff6f-446c-aafb-df8e049af2db" path="/var/lib/kubelet/pods/0971a37b-ff6f-446c-aafb-df8e049af2db/volumes"
Sep 16 13:16:42.201764 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:42.201722    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e344f9-jt85l"]
Sep 16 13:16:42.202346 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:42.202326    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0971a37b-ff6f-446c-aafb-df8e049af2db" containerName="payload-processing"
Sep 16 13:16:42.202435 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:42.202348    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0971a37b-ff6f-446c-aafb-df8e049af2db" containerName="payload-processing"
Sep 16 13:16:42.202435 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:42.202371    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47140d4f-0539-4afb-a426-d9df88f312e5" containerName="istio-proxy"
Sep 16 13:16:42.202435 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:42.202379    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="47140d4f-0539-4afb-a426-d9df88f312e5" containerName="istio-proxy"
Sep 16 13:16:42.202435 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:42.202390    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47140d4f-0539-4afb-a426-d9df88f312e5" containerName="istio-proxy"
Sep 16 13:16:42.202435 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:42.202398    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="47140d4f-0539-4afb-a426-d9df88f312e5" containerName="istio-proxy"
Sep 16 13:16:42.202435 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:42.202409    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02b22263-157b-4e9a-bba0-92f52aec0fa5" containerName="revoke-keys"
Sep 16 13:16:42.202435 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:42.202417    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="02b22263-157b-4e9a-bba0-92f52aec0fa5" containerName="revoke-keys"
Sep 16 13:16:42.202831 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:42.202530    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0971a37b-ff6f-446c-aafb-df8e049af2db" containerName="payload-processing"
Sep 16 13:16:42.202831 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:42.202542    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="47140d4f-0539-4afb-a426-d9df88f312e5" containerName="istio-proxy"
Sep 16 13:16:42.202831 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:42.202556    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="02b22263-157b-4e9a-bba0-92f52aec0fa5" containerName="revoke-keys"
Sep 16 13:16:42.202831 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:42.202568    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="47140d4f-0539-4afb-a426-d9df88f312e5" containerName="istio-proxy"
Sep 16 13:16:42.207367 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:42.207342    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e344f9-jt85l"
Sep 16 13:16:42.209962 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:42.209934    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-64zph\""
Sep 16 13:16:42.211133 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:42.211108    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 13:16:42.212999 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:42.212973    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e344f9-jt85l"]
Sep 16 13:16:42.271452 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:42.271412    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f390a202-1a5e-4743-a821-509db5bacfe0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e344f9-jt85l\" (UID: \"f390a202-1a5e-4743-a821-509db5bacfe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e344f9-jt85l"
Sep 16 13:16:42.371972 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:42.371940    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f390a202-1a5e-4743-a821-509db5bacfe0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e344f9-jt85l\" (UID: \"f390a202-1a5e-4743-a821-509db5bacfe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e344f9-jt85l"
Sep 16 13:16:42.372612 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:42.372595    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f390a202-1a5e-4743-a821-509db5bacfe0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e344f9-jt85l\" (UID: \"f390a202-1a5e-4743-a821-509db5bacfe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e344f9-jt85l"
Sep 16 13:16:42.519758 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:42.519726    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e344f9-jt85l"
Sep 16 13:16:42.794881 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:42.794850    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e344f9-jt85l"]
Sep 16 13:16:42.800124 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:16:42.800091    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf390a202_1a5e_4743_a821_509db5bacfe0.slice/crio-4a25ca8098ef650e56d49b93fc25c8e483b80237c4f80d48f10626020cc8bb89 WatchSource:0}: Error finding container 4a25ca8098ef650e56d49b93fc25c8e483b80237c4f80d48f10626020cc8bb89: Status 404 returned error can't find the container with id 4a25ca8098ef650e56d49b93fc25c8e483b80237c4f80d48f10626020cc8bb89
Sep 16 13:16:43.084780 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:43.084688    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e344f9-jt85l" event={"ID":"f390a202-1a5e-4743-a821-509db5bacfe0","Type":"ContainerStarted","Data":"63757cc9e04f732e0aa949d18eaa370855895b3064becad385a58b500828f19e"}
Sep 16 13:16:43.084780 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:43.084742    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e344f9-jt85l" event={"ID":"f390a202-1a5e-4743-a821-509db5bacfe0","Type":"ContainerStarted","Data":"4a25ca8098ef650e56d49b93fc25c8e483b80237c4f80d48f10626020cc8bb89"}
Sep 16 13:16:43.104763 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:43.104716    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e344f9-jt85l" podStartSLOduration=1.104702524 podStartE2EDuration="1.104702524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:16:43.102158826 +0000 UTC m=+2309.549391470" watchObservedRunningTime="2026-09-16 13:16:43.104702524 +0000 UTC m=+2309.551935166"
Sep 16 13:16:44.091585 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:44.091547    2577 generic.go:356] "Generic (PLEG): container finished" podID="f390a202-1a5e-4743-a821-509db5bacfe0" containerID="63757cc9e04f732e0aa949d18eaa370855895b3064becad385a58b500828f19e" exitCode=0
Sep 16 13:16:44.091953 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:44.091618    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e344f9-jt85l" event={"ID":"f390a202-1a5e-4743-a821-509db5bacfe0","Type":"ContainerDied","Data":"63757cc9e04f732e0aa949d18eaa370855895b3064becad385a58b500828f19e"}
Sep 16 13:16:45.241648 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:45.241622    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e344f9-jt85l"
Sep 16 13:16:45.302851 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:45.302820    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f390a202-1a5e-4743-a821-509db5bacfe0-maas-api-service-ca\") pod \"f390a202-1a5e-4743-a821-509db5bacfe0\" (UID: \"f390a202-1a5e-4743-a821-509db5bacfe0\") "
Sep 16 13:16:45.303180 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:45.303158    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f390a202-1a5e-4743-a821-509db5bacfe0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f390a202-1a5e-4743-a821-509db5bacfe0" (UID: "f390a202-1a5e-4743-a821-509db5bacfe0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:16:45.404630 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:45.404554    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f390a202-1a5e-4743-a821-509db5bacfe0-maas-api-service-ca\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:16:46.107283 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:46.107248    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e344f9-jt85l" event={"ID":"f390a202-1a5e-4743-a821-509db5bacfe0","Type":"ContainerDied","Data":"4a25ca8098ef650e56d49b93fc25c8e483b80237c4f80d48f10626020cc8bb89"}
Sep 16 13:16:46.107283 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:46.107267    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e344f9-jt85l"
Sep 16 13:16:46.107531 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:46.107298    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a25ca8098ef650e56d49b93fc25c8e483b80237c4f80d48f10626020cc8bb89"
Sep 16 13:16:52.969053 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:52.968945    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e344f9-29826075-ftnsd"]
Sep 16 13:16:52.974280 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:52.974250    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e344f9-29826075-ftnsd"]
Sep 16 13:16:53.007326 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:53.007293    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e344f9-85bcc9d6c-p4xzc"]
Sep 16 13:16:53.007580 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:53.007537    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e344f9-85bcc9d6c-p4xzc" podUID="8258857c-15b7-443d-91e2-97e0ed2eb413" containerName="payload-processing" containerID="cri-o://32ec150b106c8675f49b2a35943967678d1d3cd0afbb6d56c394626ed69e29af" gracePeriod=30
Sep 16 13:16:53.148051 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:53.147999    2577 generic.go:356] "Generic (PLEG): container finished" podID="8258857c-15b7-443d-91e2-97e0ed2eb413" containerID="32ec150b106c8675f49b2a35943967678d1d3cd0afbb6d56c394626ed69e29af" exitCode=0
Sep 16 13:16:53.148234 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:53.148064    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e344f9-85bcc9d6c-p4xzc" event={"ID":"8258857c-15b7-443d-91e2-97e0ed2eb413","Type":"ContainerDied","Data":"32ec150b106c8675f49b2a35943967678d1d3cd0afbb6d56c394626ed69e29af"}
Sep 16 13:16:53.270638 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:53.270614    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e344f9-85bcc9d6c-p4xzc"
Sep 16 13:16:53.300327 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:53.300294    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e344f9-jt85l"]
Sep 16 13:16:53.310257 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:53.310222    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e344f9-jt85l"]
Sep 16 13:16:53.377153 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:53.377120    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8258857c-15b7-443d-91e2-97e0ed2eb413-plugins-config-volume\") pod \"8258857c-15b7-443d-91e2-97e0ed2eb413\" (UID: \"8258857c-15b7-443d-91e2-97e0ed2eb413\") "
Sep 16 13:16:53.377345 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:53.377220    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkhv8\" (UniqueName: \"kubernetes.io/projected/8258857c-15b7-443d-91e2-97e0ed2eb413-kube-api-access-tkhv8\") pod \"8258857c-15b7-443d-91e2-97e0ed2eb413\" (UID: \"8258857c-15b7-443d-91e2-97e0ed2eb413\") "
Sep 16 13:16:53.377529 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:53.377506    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8258857c-15b7-443d-91e2-97e0ed2eb413-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8258857c-15b7-443d-91e2-97e0ed2eb413" (UID: "8258857c-15b7-443d-91e2-97e0ed2eb413"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:16:53.379421 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:53.379386    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8258857c-15b7-443d-91e2-97e0ed2eb413-kube-api-access-tkhv8" (OuterVolumeSpecName: "kube-api-access-tkhv8") pod "8258857c-15b7-443d-91e2-97e0ed2eb413" (UID: "8258857c-15b7-443d-91e2-97e0ed2eb413"). InnerVolumeSpecName "kube-api-access-tkhv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:16:53.478633 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:53.478592    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8258857c-15b7-443d-91e2-97e0ed2eb413-plugins-config-volume\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:16:53.478633 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:53.478625    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkhv8\" (UniqueName: \"kubernetes.io/projected/8258857c-15b7-443d-91e2-97e0ed2eb413-kube-api-access-tkhv8\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:16:54.055737 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:54.055704    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7db61f0f-b90d-4171-8b6b-d30d542ca9a3" path="/var/lib/kubelet/pods/7db61f0f-b90d-4171-8b6b-d30d542ca9a3/volumes"
Sep 16 13:16:54.056313 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:54.056041    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f390a202-1a5e-4743-a821-509db5bacfe0" path="/var/lib/kubelet/pods/f390a202-1a5e-4743-a821-509db5bacfe0/volumes"
Sep 16 13:16:54.154657 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:54.154622    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e344f9-85bcc9d6c-p4xzc" event={"ID":"8258857c-15b7-443d-91e2-97e0ed2eb413","Type":"ContainerDied","Data":"096ac588861cb7c3f3eda02de2ec0a7b816649c570aa5830876b9bdbc1a10ee7"}
Sep 16 13:16:54.154657 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:54.154658    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e344f9-85bcc9d6c-p4xzc"
Sep 16 13:16:54.154942 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:54.154679    2577 scope.go:117] "RemoveContainer" containerID="32ec150b106c8675f49b2a35943967678d1d3cd0afbb6d56c394626ed69e29af"
Sep 16 13:16:54.183050 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:54.183009    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e344f9-85bcc9d6c-p4xzc"]
Sep 16 13:16:54.184769 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:54.184745    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e344f9-85bcc9d6c-p4xzc"]
Sep 16 13:16:56.055297 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:16:56.055263    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8258857c-15b7-443d-91e2-97e0ed2eb413" path="/var/lib/kubelet/pods/8258857c-15b7-443d-91e2-97e0ed2eb413/volumes"
Sep 16 13:18:14.263943 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:18:14.263916    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vbt8d_0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174/console-operator/1.log"
Sep 16 13:18:14.268946 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:18:14.268925    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vbt8d_0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174/console-operator/1.log"
Sep 16 13:19:44.815396 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:19:44.815358    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s9p8x"]
Sep 16 13:19:44.815957 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:19:44.815611    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s9p8x" podUID="70656ca4-0c05-44fd-8cdb-f29a8cf2d722" containerName="manager" containerID="cri-o://3a9902c30c0b87174efab44511d404e2cc33184f6f2b6ee3c6daa72e37d9244d" gracePeriod=10
Sep 16 13:19:45.078164 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:19:45.078097    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s9p8x"
Sep 16 13:19:45.097819 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:19:45.097779    2577 generic.go:356] "Generic (PLEG): container finished" podID="70656ca4-0c05-44fd-8cdb-f29a8cf2d722" containerID="3a9902c30c0b87174efab44511d404e2cc33184f6f2b6ee3c6daa72e37d9244d" exitCode=0
Sep 16 13:19:45.097984 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:19:45.097827    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s9p8x"
Sep 16 13:19:45.097984 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:19:45.097862    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s9p8x" event={"ID":"70656ca4-0c05-44fd-8cdb-f29a8cf2d722","Type":"ContainerDied","Data":"3a9902c30c0b87174efab44511d404e2cc33184f6f2b6ee3c6daa72e37d9244d"}
Sep 16 13:19:45.097984 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:19:45.097919    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s9p8x" event={"ID":"70656ca4-0c05-44fd-8cdb-f29a8cf2d722","Type":"ContainerDied","Data":"3119c7dbfa133a22689704dcca82ec0c2bc44d1bb6e84660af7e8690d7115c83"}
Sep 16 13:19:45.097984 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:19:45.097942    2577 scope.go:117] "RemoveContainer" containerID="3a9902c30c0b87174efab44511d404e2cc33184f6f2b6ee3c6daa72e37d9244d"
Sep 16 13:19:45.110141 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:19:45.109646    2577 scope.go:117] "RemoveContainer" containerID="3a9902c30c0b87174efab44511d404e2cc33184f6f2b6ee3c6daa72e37d9244d"
Sep 16 13:19:45.110141 ip-10-0-133-190 kubenswrapper[2577]: E0916 13:19:45.110061    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a9902c30c0b87174efab44511d404e2cc33184f6f2b6ee3c6daa72e37d9244d\": container with ID starting with 3a9902c30c0b87174efab44511d404e2cc33184f6f2b6ee3c6daa72e37d9244d not found: ID does not exist" containerID="3a9902c30c0b87174efab44511d404e2cc33184f6f2b6ee3c6daa72e37d9244d"
Sep 16 13:19:45.110141 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:19:45.110098    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a9902c30c0b87174efab44511d404e2cc33184f6f2b6ee3c6daa72e37d9244d"} err="failed to get container status \"3a9902c30c0b87174efab44511d404e2cc33184f6f2b6ee3c6daa72e37d9244d\": rpc error: code = NotFound desc = could not find container \"3a9902c30c0b87174efab44511d404e2cc33184f6f2b6ee3c6daa72e37d9244d\": container with ID starting with 3a9902c30c0b87174efab44511d404e2cc33184f6f2b6ee3c6daa72e37d9244d not found: ID does not exist"
Sep 16 13:19:45.176527 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:19:45.176493    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wxdw\" (UniqueName: \"kubernetes.io/projected/70656ca4-0c05-44fd-8cdb-f29a8cf2d722-kube-api-access-2wxdw\") pod \"70656ca4-0c05-44fd-8cdb-f29a8cf2d722\" (UID: \"70656ca4-0c05-44fd-8cdb-f29a8cf2d722\") "
Sep 16 13:19:45.176699 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:19:45.176593    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70656ca4-0c05-44fd-8cdb-f29a8cf2d722-extensions-socket-volume\") pod \"70656ca4-0c05-44fd-8cdb-f29a8cf2d722\" (UID: \"70656ca4-0c05-44fd-8cdb-f29a8cf2d722\") "
Sep 16 13:19:45.176974 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:19:45.176949    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70656ca4-0c05-44fd-8cdb-f29a8cf2d722-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "70656ca4-0c05-44fd-8cdb-f29a8cf2d722" (UID: "70656ca4-0c05-44fd-8cdb-f29a8cf2d722"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:19:45.178709 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:19:45.178691    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70656ca4-0c05-44fd-8cdb-f29a8cf2d722-kube-api-access-2wxdw" (OuterVolumeSpecName: "kube-api-access-2wxdw") pod "70656ca4-0c05-44fd-8cdb-f29a8cf2d722" (UID: "70656ca4-0c05-44fd-8cdb-f29a8cf2d722"). InnerVolumeSpecName "kube-api-access-2wxdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:19:45.278254 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:19:45.278215    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wxdw\" (UniqueName: \"kubernetes.io/projected/70656ca4-0c05-44fd-8cdb-f29a8cf2d722-kube-api-access-2wxdw\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:19:45.278254 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:19:45.278249    2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70656ca4-0c05-44fd-8cdb-f29a8cf2d722-extensions-socket-volume\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:19:45.420618 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:19:45.420583    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s9p8x"]
Sep 16 13:19:45.426444 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:19:45.426414    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s9p8x"]
Sep 16 13:19:46.055441 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:19:46.055405    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70656ca4-0c05-44fd-8cdb-f29a8cf2d722" path="/var/lib/kubelet/pods/70656ca4-0c05-44fd-8cdb-f29a8cf2d722/volumes"
Sep 16 13:20:45.523139 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:20:45.523103    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr79s"]
Sep 16 13:20:45.523603 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:20:45.523471    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70656ca4-0c05-44fd-8cdb-f29a8cf2d722" containerName="manager"
Sep 16 13:20:45.523603 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:20:45.523481    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="70656ca4-0c05-44fd-8cdb-f29a8cf2d722" containerName="manager"
Sep 16 13:20:45.523603 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:20:45.523495    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8258857c-15b7-443d-91e2-97e0ed2eb413" containerName="payload-processing"
Sep 16 13:20:45.523603 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:20:45.523501    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8258857c-15b7-443d-91e2-97e0ed2eb413" containerName="payload-processing"
Sep 16 13:20:45.523603 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:20:45.523510    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f390a202-1a5e-4743-a821-509db5bacfe0" containerName="revoke-keys"
Sep 16 13:20:45.523603 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:20:45.523515    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f390a202-1a5e-4743-a821-509db5bacfe0" containerName="revoke-keys"
Sep 16 13:20:45.523603 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:20:45.523577    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8258857c-15b7-443d-91e2-97e0ed2eb413" containerName="payload-processing"
Sep 16 13:20:45.523603 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:20:45.523587    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f390a202-1a5e-4743-a821-509db5bacfe0" containerName="revoke-keys"
Sep 16 13:20:45.523603 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:20:45.523592    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="70656ca4-0c05-44fd-8cdb-f29a8cf2d722" containerName="manager"
Sep 16 13:20:45.526588 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:20:45.526570    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr79s"
Sep 16 13:20:45.529642 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:20:45.529620    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-hsr4g\""
Sep 16 13:20:45.543992 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:20:45.543965    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr79s"]
Sep 16 13:20:45.715635 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:20:45.715594    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5vcx\" (UniqueName: \"kubernetes.io/projected/7233ff48-7428-403a-87b7-7f5920020b09-kube-api-access-c5vcx\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xr79s\" (UID: \"7233ff48-7428-403a-87b7-7f5920020b09\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr79s"
Sep 16 13:20:45.715825 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:20:45.715644    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7233ff48-7428-403a-87b7-7f5920020b09-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xr79s\" (UID: \"7233ff48-7428-403a-87b7-7f5920020b09\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr79s"
Sep 16 13:20:45.816347 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:20:45.816259    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5vcx\" (UniqueName: \"kubernetes.io/projected/7233ff48-7428-403a-87b7-7f5920020b09-kube-api-access-c5vcx\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xr79s\" (UID: \"7233ff48-7428-403a-87b7-7f5920020b09\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr79s"
Sep 16 13:20:45.816347 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:20:45.816313    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7233ff48-7428-403a-87b7-7f5920020b09-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xr79s\" (UID: \"7233ff48-7428-403a-87b7-7f5920020b09\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr79s"
Sep 16 13:20:45.816665 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:20:45.816647    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7233ff48-7428-403a-87b7-7f5920020b09-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xr79s\" (UID: \"7233ff48-7428-403a-87b7-7f5920020b09\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr79s"
Sep 16 13:20:45.824693 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:20:45.824671    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5vcx\" (UniqueName: \"kubernetes.io/projected/7233ff48-7428-403a-87b7-7f5920020b09-kube-api-access-c5vcx\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xr79s\" (UID: \"7233ff48-7428-403a-87b7-7f5920020b09\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr79s"
Sep 16 13:20:45.837520 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:20:45.837494    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr79s"
Sep 16 13:20:46.122702 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:20:46.122675    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr79s"]
Sep 16 13:20:46.123732 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:20:46.123711    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7233ff48_7428_403a_87b7_7f5920020b09.slice/crio-2659e6773119c75b754b666571ca827a04325daf7ace9535ca8265fde3b24400 WatchSource:0}: Error finding container 2659e6773119c75b754b666571ca827a04325daf7ace9535ca8265fde3b24400: Status 404 returned error can't find the container with id 2659e6773119c75b754b666571ca827a04325daf7ace9535ca8265fde3b24400
Sep 16 13:20:46.423569 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:20:46.423480    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr79s" event={"ID":"7233ff48-7428-403a-87b7-7f5920020b09","Type":"ContainerStarted","Data":"5a132848211007fd5bd33816c1091ec97aba2d209fc594b5e35f263aca88527f"}
Sep 16 13:20:46.423569 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:20:46.423543    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr79s" event={"ID":"7233ff48-7428-403a-87b7-7f5920020b09","Type":"ContainerStarted","Data":"2659e6773119c75b754b666571ca827a04325daf7ace9535ca8265fde3b24400"}
Sep 16 13:20:46.423759 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:20:46.423689    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr79s"
Sep 16 13:20:46.445083 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:20:46.444996    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr79s" podStartSLOduration=1.444977367 podStartE2EDuration="1.444977367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:20:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:20:46.443160677 +0000 UTC m=+2552.890393331" watchObservedRunningTime="2026-09-16 13:20:46.444977367 +0000 UTC m=+2552.892210010"
Sep 16 13:20:57.432388 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:20:57.432354    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr79s"
Sep 16 13:21:15.307168 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:21:15.307136    2577 scope.go:117] "RemoveContainer" containerID="d581be2322b6e0a620691ff7c137071df46f69eea7e29564f2dcaf434b5fdafd"
Sep 16 13:21:15.316471 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:21:15.316446    2577 scope.go:117] "RemoveContainer" containerID="b7d4c126abef7775a200589bd2a36f016fea69eac6b717085bb478edadca6a80"
Sep 16 13:21:15.324881 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:21:15.324862    2577 scope.go:117] "RemoveContainer" containerID="76d1bc286f1cc3e691338c5a720394047874319cbd634ac3fd72304044d294b8"
Sep 16 13:21:15.333001 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:21:15.332983    2577 scope.go:117] "RemoveContainer" containerID="7996270a2088626ab38d47793892fc34dcfe6a8a0f406072817d6a8d417ba8d8"
Sep 16 13:21:36.545618 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:21:36.545529    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24"]
Sep 16 13:21:36.549259 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:21:36.549235    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24"
Sep 16 13:21:36.552009 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:21:36.551988    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-00d58807-kserve-self-signed-certs\""
Sep 16 13:21:36.562545 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:21:36.562521    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24"]
Sep 16 13:21:36.586603 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:21:36.586572    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dfl9\" (UniqueName: \"kubernetes.io/projected/0a4abffb-5392-4a8a-9dc3-782a92e708c9-kube-api-access-8dfl9\") pod \"e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24\" (UID: \"0a4abffb-5392-4a8a-9dc3-782a92e708c9\") " pod="llm/e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24"
Sep 16 13:21:36.586742 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:21:36.586614    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a4abffb-5392-4a8a-9dc3-782a92e708c9-model-cache\") pod \"e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24\" (UID: \"0a4abffb-5392-4a8a-9dc3-782a92e708c9\") " pod="llm/e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24"
Sep 16 13:21:36.586742 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:21:36.586640    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a4abffb-5392-4a8a-9dc3-782a92e708c9-tmp-dir\") pod \"e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24\" (UID: \"0a4abffb-5392-4a8a-9dc3-782a92e708c9\") " pod="llm/e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24"
Sep 16 13:21:36.586742 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:21:36.586684    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a4abffb-5392-4a8a-9dc3-782a92e708c9-home\") pod \"e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24\" (UID: \"0a4abffb-5392-4a8a-9dc3-782a92e708c9\") " pod="llm/e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24"
Sep 16 13:21:36.586742 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:21:36.586704    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a4abffb-5392-4a8a-9dc3-782a92e708c9-dshm\") pod \"e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24\" (UID: \"0a4abffb-5392-4a8a-9dc3-782a92e708c9\") " pod="llm/e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24"
Sep 16 13:21:36.586742 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:21:36.586721    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a4abffb-5392-4a8a-9dc3-782a92e708c9-tls-certs\") pod \"e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24\" (UID: \"0a4abffb-5392-4a8a-9dc3-782a92e708c9\") " pod="llm/e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24"
Sep 16 13:21:36.687814 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:21:36.687777    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dfl9\" (UniqueName: \"kubernetes.io/projected/0a4abffb-5392-4a8a-9dc3-782a92e708c9-kube-api-access-8dfl9\") pod \"e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24\" (UID: \"0a4abffb-5392-4a8a-9dc3-782a92e708c9\") " pod="llm/e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24"
Sep 16 13:21:36.688008 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:21:36.687842    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a4abffb-5392-4a8a-9dc3-782a92e708c9-model-cache\") pod \"e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24\" (UID: \"0a4abffb-5392-4a8a-9dc3-782a92e708c9\") " pod="llm/e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24"
Sep 16 13:21:36.688008 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:21:36.687884    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a4abffb-5392-4a8a-9dc3-782a92e708c9-tmp-dir\") pod \"e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24\" (UID: \"0a4abffb-5392-4a8a-9dc3-782a92e708c9\") " pod="llm/e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24"
Sep 16 13:21:36.688008 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:21:36.687912    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a4abffb-5392-4a8a-9dc3-782a92e708c9-home\") pod \"e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24\" (UID: \"0a4abffb-5392-4a8a-9dc3-782a92e708c9\") " pod="llm/e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24"
Sep 16 13:21:36.688229 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:21:36.688080    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a4abffb-5392-4a8a-9dc3-782a92e708c9-dshm\") pod \"e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24\" (UID: \"0a4abffb-5392-4a8a-9dc3-782a92e708c9\") " pod="llm/e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24"
Sep 16 13:21:36.688229 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:21:36.688129    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a4abffb-5392-4a8a-9dc3-782a92e708c9-tls-certs\") pod \"e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24\" (UID: \"0a4abffb-5392-4a8a-9dc3-782a92e708c9\") " pod="llm/e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24"
Sep 16 13:21:36.688378 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:21:36.688359    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a4abffb-5392-4a8a-9dc3-782a92e708c9-tmp-dir\") pod \"e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24\" (UID: \"0a4abffb-5392-4a8a-9dc3-782a92e708c9\") " pod="llm/e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24"
Sep 16 13:21:36.688442 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:21:36.688389    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a4abffb-5392-4a8a-9dc3-782a92e708c9-home\") pod \"e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24\" (UID: \"0a4abffb-5392-4a8a-9dc3-782a92e708c9\") " pod="llm/e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24"
Sep 16 13:21:36.688763 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:21:36.688741    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a4abffb-5392-4a8a-9dc3-782a92e708c9-model-cache\") pod \"e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24\" (UID: \"0a4abffb-5392-4a8a-9dc3-782a92e708c9\") " pod="llm/e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24"
Sep 16 13:21:36.690576 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:21:36.690552    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a4abffb-5392-4a8a-9dc3-782a92e708c9-dshm\") pod \"e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24\" (UID: \"0a4abffb-5392-4a8a-9dc3-782a92e708c9\") " pod="llm/e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24"
Sep 16 13:21:36.691257 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:21:36.691233    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a4abffb-5392-4a8a-9dc3-782a92e708c9-tls-certs\") pod \"e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24\" (UID: \"0a4abffb-5392-4a8a-9dc3-782a92e708c9\") " pod="llm/e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24"
Sep 16 13:21:36.700517 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:21:36.700491    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dfl9\" (UniqueName: \"kubernetes.io/projected/0a4abffb-5392-4a8a-9dc3-782a92e708c9-kube-api-access-8dfl9\") pod \"e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24\" (UID: \"0a4abffb-5392-4a8a-9dc3-782a92e708c9\") " pod="llm/e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24"
Sep 16 13:21:36.858445 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:21:36.858364    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24"
Sep 16 13:21:37.133534 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:21:37.133512    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24"]
Sep 16 13:21:37.135756 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:21:37.135731    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a4abffb_5392_4a8a_9dc3_782a92e708c9.slice/crio-33b4d550a9c94547a627b3da16de1bf7bd0c9adb5ac3e0df75ab7e47074e5b91 WatchSource:0}: Error finding container 33b4d550a9c94547a627b3da16de1bf7bd0c9adb5ac3e0df75ab7e47074e5b91: Status 404 returned error can't find the container with id 33b4d550a9c94547a627b3da16de1bf7bd0c9adb5ac3e0df75ab7e47074e5b91
Sep 16 13:21:37.720853 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:21:37.720815    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24" event={"ID":"0a4abffb-5392-4a8a-9dc3-782a92e708c9","Type":"ContainerStarted","Data":"ab8a1110833f0b854148f11d80d4848ea45b0b45c1705e6ddc4c123709b37cc3"}
Sep 16 13:21:37.721278 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:21:37.720867    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24" event={"ID":"0a4abffb-5392-4a8a-9dc3-782a92e708c9","Type":"ContainerStarted","Data":"33b4d550a9c94547a627b3da16de1bf7bd0c9adb5ac3e0df75ab7e47074e5b91"}
Sep 16 13:21:37.741215 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:21:37.741162    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24" podStartSLOduration=1.741145023 podStartE2EDuration="1.741145023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:21:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:21:37.738366833 +0000 UTC m=+2604.185599487" watchObservedRunningTime="2026-09-16 13:21:37.741145023 +0000 UTC m=+2604.188377664"
Sep 16 13:21:46.859241 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:21:46.859196    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24"
Sep 16 13:21:46.859241 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:21:46.859249    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24"
Sep 16 13:21:46.872521 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:21:46.872491    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24"
Sep 16 13:21:47.791999 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:21:47.791965    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24"
Sep 16 13:22:15.362726 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:22:15.362694    2577 scope.go:117] "RemoveContainer" containerID="515f701a94b5630b922e2e9f73d9d5476e2722d98d8f37885d5de8d6d7db81df"
Sep 16 13:22:35.879596 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:22:35.879561    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24"]
Sep 16 13:22:50.899928 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:22:50.899861    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24" podUID="0a4abffb-5392-4a8a-9dc3-782a92e708c9" containerName="main" containerID="cri-o://ab8a1110833f0b854148f11d80d4848ea45b0b45c1705e6ddc4c123709b37cc3" gracePeriod=45
Sep 16 13:22:51.115511 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:22:51.115476    2577 generic.go:356] "Generic (PLEG): container finished" podID="0a4abffb-5392-4a8a-9dc3-782a92e708c9" containerID="ab8a1110833f0b854148f11d80d4848ea45b0b45c1705e6ddc4c123709b37cc3" exitCode=0
Sep 16 13:22:51.115685 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:22:51.115553    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24" event={"ID":"0a4abffb-5392-4a8a-9dc3-782a92e708c9","Type":"ContainerDied","Data":"ab8a1110833f0b854148f11d80d4848ea45b0b45c1705e6ddc4c123709b37cc3"}
Sep 16 13:22:51.155628 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:22:51.155570    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24"
Sep 16 13:22:51.287662 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:22:51.287634    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a4abffb-5392-4a8a-9dc3-782a92e708c9-tmp-dir\") pod \"0a4abffb-5392-4a8a-9dc3-782a92e708c9\" (UID: \"0a4abffb-5392-4a8a-9dc3-782a92e708c9\") "
Sep 16 13:22:51.287866 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:22:51.287679    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a4abffb-5392-4a8a-9dc3-782a92e708c9-tls-certs\") pod \"0a4abffb-5392-4a8a-9dc3-782a92e708c9\" (UID: \"0a4abffb-5392-4a8a-9dc3-782a92e708c9\") "
Sep 16 13:22:51.287866 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:22:51.287754    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dfl9\" (UniqueName: \"kubernetes.io/projected/0a4abffb-5392-4a8a-9dc3-782a92e708c9-kube-api-access-8dfl9\") pod \"0a4abffb-5392-4a8a-9dc3-782a92e708c9\" (UID: \"0a4abffb-5392-4a8a-9dc3-782a92e708c9\") "
Sep 16 13:22:51.287866 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:22:51.287800    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a4abffb-5392-4a8a-9dc3-782a92e708c9-dshm\") pod \"0a4abffb-5392-4a8a-9dc3-782a92e708c9\" (UID: \"0a4abffb-5392-4a8a-9dc3-782a92e708c9\") "
Sep 16 13:22:51.287866 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:22:51.287839    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a4abffb-5392-4a8a-9dc3-782a92e708c9-home\") pod \"0a4abffb-5392-4a8a-9dc3-782a92e708c9\" (UID: \"0a4abffb-5392-4a8a-9dc3-782a92e708c9\") "
Sep 16 13:22:51.288100 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:22:51.287885    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a4abffb-5392-4a8a-9dc3-782a92e708c9-model-cache\") pod \"0a4abffb-5392-4a8a-9dc3-782a92e708c9\" (UID: \"0a4abffb-5392-4a8a-9dc3-782a92e708c9\") "
Sep 16 13:22:51.288100 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:22:51.287938    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a4abffb-5392-4a8a-9dc3-782a92e708c9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0a4abffb-5392-4a8a-9dc3-782a92e708c9" (UID: "0a4abffb-5392-4a8a-9dc3-782a92e708c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:22:51.288205 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:22:51.288191    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a4abffb-5392-4a8a-9dc3-782a92e708c9-tmp-dir\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:22:51.288330 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:22:51.288306    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a4abffb-5392-4a8a-9dc3-782a92e708c9-home" (OuterVolumeSpecName: "home") pod "0a4abffb-5392-4a8a-9dc3-782a92e708c9" (UID: "0a4abffb-5392-4a8a-9dc3-782a92e708c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:22:51.288403 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:22:51.288385    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a4abffb-5392-4a8a-9dc3-782a92e708c9-model-cache" (OuterVolumeSpecName: "model-cache") pod "0a4abffb-5392-4a8a-9dc3-782a92e708c9" (UID: "0a4abffb-5392-4a8a-9dc3-782a92e708c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:22:51.290689 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:22:51.290665    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a4abffb-5392-4a8a-9dc3-782a92e708c9-dshm" (OuterVolumeSpecName: "dshm") pod "0a4abffb-5392-4a8a-9dc3-782a92e708c9" (UID: "0a4abffb-5392-4a8a-9dc3-782a92e708c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:22:51.290689 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:22:51.290675    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a4abffb-5392-4a8a-9dc3-782a92e708c9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0a4abffb-5392-4a8a-9dc3-782a92e708c9" (UID: "0a4abffb-5392-4a8a-9dc3-782a92e708c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:22:51.290689 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:22:51.290686    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a4abffb-5392-4a8a-9dc3-782a92e708c9-kube-api-access-8dfl9" (OuterVolumeSpecName: "kube-api-access-8dfl9") pod "0a4abffb-5392-4a8a-9dc3-782a92e708c9" (UID: "0a4abffb-5392-4a8a-9dc3-782a92e708c9"). InnerVolumeSpecName "kube-api-access-8dfl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:22:51.389483 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:22:51.389447    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a4abffb-5392-4a8a-9dc3-782a92e708c9-tls-certs\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:22:51.389483 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:22:51.389477    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dfl9\" (UniqueName: \"kubernetes.io/projected/0a4abffb-5392-4a8a-9dc3-782a92e708c9-kube-api-access-8dfl9\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:22:51.389483 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:22:51.389491    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a4abffb-5392-4a8a-9dc3-782a92e708c9-dshm\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:22:51.389704 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:22:51.389502    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a4abffb-5392-4a8a-9dc3-782a92e708c9-home\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:22:51.389704 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:22:51.389512    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a4abffb-5392-4a8a-9dc3-782a92e708c9-model-cache\") on node \"ip-10-0-133-190.ec2.internal\" DevicePath \"\""
Sep 16 13:22:52.122262 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:22:52.122229    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24"
Sep 16 13:22:52.122262 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:22:52.122250    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24" event={"ID":"0a4abffb-5392-4a8a-9dc3-782a92e708c9","Type":"ContainerDied","Data":"33b4d550a9c94547a627b3da16de1bf7bd0c9adb5ac3e0df75ab7e47074e5b91"}
Sep 16 13:22:52.122698 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:22:52.122307    2577 scope.go:117] "RemoveContainer" containerID="ab8a1110833f0b854148f11d80d4848ea45b0b45c1705e6ddc4c123709b37cc3"
Sep 16 13:22:52.140425 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:22:52.140399    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24"]
Sep 16 13:22:52.144564 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:22:52.144542    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-00d58807-kserve-5ff9c5b5cf-4fx24"]
Sep 16 13:22:54.054862 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:22:54.054830    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a4abffb-5392-4a8a-9dc3-782a92e708c9" path="/var/lib/kubelet/pods/0a4abffb-5392-4a8a-9dc3-782a92e708c9/volumes"
Sep 16 13:23:14.295109 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:14.294976    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vbt8d_0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174/console-operator/1.log"
Sep 16 13:23:14.304570 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:14.304550    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vbt8d_0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174/console-operator/1.log"
Sep 16 13:23:15.386616 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:15.386581    2577 scope.go:117] "RemoveContainer" containerID="854c3faf6a990497690575ff98fe21baa27aa1bdab8c0b1232998192ac4ec4c1"
Sep 16 13:23:15.399381 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:15.399357    2577 scope.go:117] "RemoveContainer" containerID="63757cc9e04f732e0aa949d18eaa370855895b3064becad385a58b500828f19e"
Sep 16 13:23:32.674711 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:32.674676    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-ssfwx_9ed52a74-3e4e-4519-b070-f1bca5653647/manager/0.log"
Sep 16 13:23:33.292000 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:33.291967    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c7bc95455-h5zdt_ad54ab6a-787b-471a-a648-1badf8c1b94c/manager/0.log"
Sep 16 13:23:33.298715 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:33.298695    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c7bc95455-h5zdt_ad54ab6a-787b-471a-a648-1badf8c1b94c/copy-manifests/0.log"
Sep 16 13:23:34.102881 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:34.102849    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29826075-mtpck_a6396d36-b7ed-4c46-9ce9-c6cc3608e570/cleanup/0.log"
Sep 16 13:23:35.227400 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:35.227373    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-wwn5q_96b6599b-adb3-4b7c-b3fa-df54c38b0f00/manager/0.log"
Sep 16 13:23:35.351591 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:35.351562    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-77dd6_463edcc3-887a-4637-9702-bc62f152f9ff/manager/0.log"
Sep 16 13:23:35.618665 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:35.618631    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-xr79s_7233ff48-7428-403a-87b7-7f5920020b09/manager/0.log"
Sep 16 13:23:36.226204 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:36.226175    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s_ef53d612-0d2e-426f-a926-bcee1b7d4827/istio-proxy/0.log"
Sep 16 13:23:36.707752 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:36.707724    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9rfz6_be45e9ae-2a43-4cd9-bd86-541bc2a97291/istio-proxy/1.log"
Sep 16 13:23:37.086112 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:37.086054    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-944d5c59d-l5g2b_b04ec1dc-5e71-4fed-abc5-6875befad478/router/0.log"
Sep 16 13:23:37.776383 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:37.776348    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-4slnl_80880178-50ec-490f-a609-52efa72dcedc/main/0.log"
Sep 16 13:23:37.900824 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:37.900792    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f29wmf_e54d5f1e-0503-4b28-88f6-53cd5ac6062e/main/0.log"
Sep 16 13:23:38.021990 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:38.021961    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-bvp5c_8010ee2c-f62f-4b5a-a729-e26019b3364d/main/0.log"
Sep 16 13:23:43.944806 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:43.944774    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fs84m/must-gather-mrkb7"]
Sep 16 13:23:43.945303 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:43.945229    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a4abffb-5392-4a8a-9dc3-782a92e708c9" containerName="main"
Sep 16 13:23:43.945303 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:43.945246    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a4abffb-5392-4a8a-9dc3-782a92e708c9" containerName="main"
Sep 16 13:23:43.945458 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:43.945330    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a4abffb-5392-4a8a-9dc3-782a92e708c9" containerName="main"
Sep 16 13:23:43.948622 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:43.948602    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fs84m/must-gather-mrkb7"
Sep 16 13:23:43.952828 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:43.952798    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fs84m\"/\"default-dockercfg-rz5kq\""
Sep 16 13:23:43.952828 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:43.952805    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fs84m\"/\"openshift-service-ca.crt\""
Sep 16 13:23:43.953087 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:43.952924    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fs84m\"/\"kube-root-ca.crt\""
Sep 16 13:23:43.959122 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:43.959065    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fs84m/must-gather-mrkb7"]
Sep 16 13:23:43.978743 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:43.978712    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/18c43c75-1f07-4b60-a77c-4bb553946808-must-gather-output\") pod \"must-gather-mrkb7\" (UID: \"18c43c75-1f07-4b60-a77c-4bb553946808\") " pod="openshift-must-gather-fs84m/must-gather-mrkb7"
Sep 16 13:23:43.978895 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:43.978837    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s75ll\" (UniqueName: \"kubernetes.io/projected/18c43c75-1f07-4b60-a77c-4bb553946808-kube-api-access-s75ll\") pod \"must-gather-mrkb7\" (UID: \"18c43c75-1f07-4b60-a77c-4bb553946808\") " pod="openshift-must-gather-fs84m/must-gather-mrkb7"
Sep 16 13:23:44.079323 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:44.079291    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/18c43c75-1f07-4b60-a77c-4bb553946808-must-gather-output\") pod \"must-gather-mrkb7\" (UID: \"18c43c75-1f07-4b60-a77c-4bb553946808\") " pod="openshift-must-gather-fs84m/must-gather-mrkb7"
Sep 16 13:23:44.079555 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:44.079375    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s75ll\" (UniqueName: \"kubernetes.io/projected/18c43c75-1f07-4b60-a77c-4bb553946808-kube-api-access-s75ll\") pod \"must-gather-mrkb7\" (UID: \"18c43c75-1f07-4b60-a77c-4bb553946808\") " pod="openshift-must-gather-fs84m/must-gather-mrkb7"
Sep 16 13:23:44.079633 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:44.079612    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/18c43c75-1f07-4b60-a77c-4bb553946808-must-gather-output\") pod \"must-gather-mrkb7\" (UID: \"18c43c75-1f07-4b60-a77c-4bb553946808\") " pod="openshift-must-gather-fs84m/must-gather-mrkb7"
Sep 16 13:23:44.087590 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:44.087562    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s75ll\" (UniqueName: \"kubernetes.io/projected/18c43c75-1f07-4b60-a77c-4bb553946808-kube-api-access-s75ll\") pod \"must-gather-mrkb7\" (UID: \"18c43c75-1f07-4b60-a77c-4bb553946808\") " pod="openshift-must-gather-fs84m/must-gather-mrkb7"
Sep 16 13:23:44.258640 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:44.258615    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fs84m/must-gather-mrkb7"
Sep 16 13:23:44.532379 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:44.532354    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fs84m/must-gather-mrkb7"]
Sep 16 13:23:44.534558 ip-10-0-133-190 kubenswrapper[2577]: W0916 13:23:44.534528    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18c43c75_1f07_4b60_a77c_4bb553946808.slice/crio-481bcbafec4144c47d2f7a187267f457e1b40523e378bc468bdc9e71f5594f98 WatchSource:0}: Error finding container 481bcbafec4144c47d2f7a187267f457e1b40523e378bc468bdc9e71f5594f98: Status 404 returned error can't find the container with id 481bcbafec4144c47d2f7a187267f457e1b40523e378bc468bdc9e71f5594f98
Sep 16 13:23:44.536567 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:44.536551    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 13:23:45.408999 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:45.408964    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fs84m/must-gather-mrkb7" event={"ID":"18c43c75-1f07-4b60-a77c-4bb553946808","Type":"ContainerStarted","Data":"481bcbafec4144c47d2f7a187267f457e1b40523e378bc468bdc9e71f5594f98"}
Sep 16 13:23:46.418485 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:46.418444    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fs84m/must-gather-mrkb7" event={"ID":"18c43c75-1f07-4b60-a77c-4bb553946808","Type":"ContainerStarted","Data":"fada02f913c88965efc9609b7c32a0e2c6dafdae8381739d901bfd086cfd10ce"}
Sep 16 13:23:46.418893 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:46.418496    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fs84m/must-gather-mrkb7" event={"ID":"18c43c75-1f07-4b60-a77c-4bb553946808","Type":"ContainerStarted","Data":"144a5826411b96d4091024279e4fc7a6ffde85df32d7a823e9c2918c5dd7f6b3"}
Sep 16 13:23:46.439749 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:46.439701    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fs84m/must-gather-mrkb7" podStartSLOduration=2.594751342 podStartE2EDuration="3.439684389s" totalImagesPullingTime="844.933047ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:23:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:23:46.437096727 +0000 UTC m=+2732.884329371" watchObservedRunningTime="2026-09-16 13:23:46.439684389 +0000 UTC m=+2732.886917031"
Sep 16 13:23:46.975619 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:46.975581    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bwb7l_342f831e-0ffe-4582-b616-e0ce14364b32/global-pull-secret-syncer/0.log"
Sep 16 13:23:47.187721 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:47.187678    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-j2kjn_c9fcc2dc-c3c3-401c-8534-95cd3908332e/konnectivity-agent/0.log"
Sep 16 13:23:47.286068 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:47.286009    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-190.ec2.internal_dad4cbabbaf3126a41405ddf99389dd4/haproxy/0.log"
Sep 16 13:23:51.974117 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:51.974080    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-wwn5q_96b6599b-adb3-4b7c-b3fa-df54c38b0f00/manager/0.log"
Sep 16 13:23:52.003543 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:52.003502    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-77dd6_463edcc3-887a-4637-9702-bc62f152f9ff/manager/0.log"
Sep 16 13:23:52.081188 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:52.081160    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-xr79s_7233ff48-7428-403a-87b7-7f5920020b09/manager/0.log"
Sep 16 13:23:53.788636 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:53.788583    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-mlzkb_a583c3d2-5168-4901-89ec-2c64993e60ec/cluster-monitoring-operator/0.log"
Sep 16 13:23:53.817846 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:53.817812    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-fnszz_2b2aba6f-e269-465a-9bf6-7d1059876ffb/kube-state-metrics/0.log"
Sep 16 13:23:53.838531 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:53.838496    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-fnszz_2b2aba6f-e269-465a-9bf6-7d1059876ffb/kube-rbac-proxy-main/0.log"
Sep 16 13:23:53.865638 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:53.865600    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-fnszz_2b2aba6f-e269-465a-9bf6-7d1059876ffb/kube-rbac-proxy-self/0.log"
Sep 16 13:23:54.137924 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:54.137849    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jbh4p_d264703a-7eb5-46a8-9dc6-49c3bdfb84e0/node-exporter/0.log"
Sep 16 13:23:54.164259 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:54.164231    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jbh4p_d264703a-7eb5-46a8-9dc6-49c3bdfb84e0/kube-rbac-proxy/0.log"
Sep 16 13:23:54.191867 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:54.191824    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jbh4p_d264703a-7eb5-46a8-9dc6-49c3bdfb84e0/init-textfile/0.log"
Sep 16 13:23:54.326340 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:54.326308    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-jvdbf_aa46cae3-6f61-42f7-89c1-be418493471c/kube-rbac-proxy-main/0.log"
Sep 16 13:23:54.348404 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:54.348370    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-jvdbf_aa46cae3-6f61-42f7-89c1-be418493471c/kube-rbac-proxy-self/0.log"
Sep 16 13:23:54.376198 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:54.376172    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-jvdbf_aa46cae3-6f61-42f7-89c1-be418493471c/openshift-state-metrics/0.log"
Sep 16 13:23:54.734550 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:54.734448    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-q6gv5_cb1916e5-42a2-4beb-8deb-0420da03a475/prometheus-operator/0.log"
Sep 16 13:23:54.764328 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:54.764303    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-q6gv5_cb1916e5-42a2-4beb-8deb-0420da03a475/kube-rbac-proxy/0.log"
Sep 16 13:23:54.790349 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:54.790269    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-75qbr_ff18210a-b422-4c2d-bdd8-3332f8c0b78d/prometheus-operator-admission-webhook/0.log"
Sep 16 13:23:54.830022 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:54.829995    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8b96c8b44-w7j7n_f7d083fa-9933-493c-8caa-c1a2dae789e4/telemeter-client/0.log"
Sep 16 13:23:54.860098 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:54.860068    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8b96c8b44-w7j7n_f7d083fa-9933-493c-8caa-c1a2dae789e4/reload/0.log"
Sep 16 13:23:54.886969 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:54.886936    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8b96c8b44-w7j7n_f7d083fa-9933-493c-8caa-c1a2dae789e4/kube-rbac-proxy/0.log"
Sep 16 13:23:56.934410 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:56.934370    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fs84m/perf-node-gather-daemonset-mq764"]
Sep 16 13:23:56.941659 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:56.941632    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-mq764"
Sep 16 13:23:56.949232 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:56.949203    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fs84m/perf-node-gather-daemonset-mq764"]
Sep 16 13:23:56.999985 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:56.999951    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vbt8d_0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174/console-operator/1.log"
Sep 16 13:23:57.010127 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:57.010101    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vbt8d_0ab2d86c-9ac2-4b80-87c5-d5f6d98fb174/console-operator/2.log"
Sep 16 13:23:57.026602 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:57.026556    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a75ad2bd-3ed0-4831-ab47-94d160e3b6dc-sys\") pod \"perf-node-gather-daemonset-mq764\" (UID: \"a75ad2bd-3ed0-4831-ab47-94d160e3b6dc\") " pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-mq764"
Sep 16 13:23:57.026863 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:57.026842    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a75ad2bd-3ed0-4831-ab47-94d160e3b6dc-lib-modules\") pod \"perf-node-gather-daemonset-mq764\" (UID: \"a75ad2bd-3ed0-4831-ab47-94d160e3b6dc\") " pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-mq764"
Sep 16 13:23:57.027120 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:57.027092    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a75ad2bd-3ed0-4831-ab47-94d160e3b6dc-proc\") pod \"perf-node-gather-daemonset-mq764\" (UID: \"a75ad2bd-3ed0-4831-ab47-94d160e3b6dc\") " pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-mq764"
Sep 16 13:23:57.027271 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:57.027254    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a75ad2bd-3ed0-4831-ab47-94d160e3b6dc-podres\") pod \"perf-node-gather-daemonset-mq764\" (UID: \"a75ad2bd-3ed0-4831-ab47-94d160e3b6dc\") " pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-mq764"
Sep 16 13:23:57.027416 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:57.027400    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tbzj\" (UniqueName: \"kubernetes.io/projected/a75ad2bd-3ed0-4831-ab47-94d160e3b6dc-kube-api-access-4tbzj\") pod \"perf-node-gather-daemonset-mq764\" (UID: \"a75ad2bd-3ed0-4831-ab47-94d160e3b6dc\") " pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-mq764"
Sep 16 13:23:57.129252 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:57.129209    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a75ad2bd-3ed0-4831-ab47-94d160e3b6dc-sys\") pod \"perf-node-gather-daemonset-mq764\" (UID: \"a75ad2bd-3ed0-4831-ab47-94d160e3b6dc\") " pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-mq764"
Sep 16 13:23:57.129487 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:57.129457    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a75ad2bd-3ed0-4831-ab47-94d160e3b6dc-sys\") pod \"perf-node-gather-daemonset-mq764\" (UID: \"a75ad2bd-3ed0-4831-ab47-94d160e3b6dc\") " pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-mq764"
Sep 16 13:23:57.129613 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:57.129585    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a75ad2bd-3ed0-4831-ab47-94d160e3b6dc-lib-modules\") pod \"perf-node-gather-daemonset-mq764\" (UID: \"a75ad2bd-3ed0-4831-ab47-94d160e3b6dc\") " pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-mq764"
Sep 16 13:23:57.129931 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:57.129915    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a75ad2bd-3ed0-4831-ab47-94d160e3b6dc-proc\") pod \"perf-node-gather-daemonset-mq764\" (UID: \"a75ad2bd-3ed0-4831-ab47-94d160e3b6dc\") " pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-mq764"
Sep 16 13:23:57.130276 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:57.130159    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a75ad2bd-3ed0-4831-ab47-94d160e3b6dc-podres\") pod \"perf-node-gather-daemonset-mq764\" (UID: \"a75ad2bd-3ed0-4831-ab47-94d160e3b6dc\") " pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-mq764"
Sep 16 13:23:57.130420 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:57.130400    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tbzj\" (UniqueName: \"kubernetes.io/projected/a75ad2bd-3ed0-4831-ab47-94d160e3b6dc-kube-api-access-4tbzj\") pod \"perf-node-gather-daemonset-mq764\" (UID: \"a75ad2bd-3ed0-4831-ab47-94d160e3b6dc\") " pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-mq764"
Sep 16 13:23:57.131335 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:57.130263    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a75ad2bd-3ed0-4831-ab47-94d160e3b6dc-podres\") pod \"perf-node-gather-daemonset-mq764\" (UID: \"a75ad2bd-3ed0-4831-ab47-94d160e3b6dc\") " pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-mq764"
Sep 16 13:23:57.131474 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:57.129816    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a75ad2bd-3ed0-4831-ab47-94d160e3b6dc-lib-modules\") pod \"perf-node-gather-daemonset-mq764\" (UID: \"a75ad2bd-3ed0-4831-ab47-94d160e3b6dc\") " pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-mq764"
Sep 16 13:23:57.131589 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:57.130100    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a75ad2bd-3ed0-4831-ab47-94d160e3b6dc-proc\") pod \"perf-node-gather-daemonset-mq764\" (UID: \"a75ad2bd-3ed0-4831-ab47-94d160e3b6dc\") " pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-mq764"
Sep 16 13:23:57.140907 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:57.140876    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tbzj\" (UniqueName: \"kubernetes.io/projected/a75ad2bd-3ed0-4831-ab47-94d160e3b6dc-kube-api-access-4tbzj\") pod \"perf-node-gather-daemonset-mq764\" (UID: \"a75ad2bd-3ed0-4831-ab47-94d160e3b6dc\") " pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-mq764"
Sep 16 13:23:57.263481 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:57.263440    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-mq764"
Sep 16 13:23:57.569587 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:57.566526    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fs84m/perf-node-gather-daemonset-mq764"]
Sep 16 13:23:57.615967 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:57.615941    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-55c45d8cc9-48v4z_ca507687-8e56-4d00-8bd1-9aa1b80d489b/console/0.log"
Sep 16 13:23:57.663219 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:57.663111    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-kl8tl_c41aa457-55d7-4f45-a1b8-f42d4597182c/download-server/0.log"
Sep 16 13:23:58.397506 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:58.397464    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-mhv8x_e0992b2d-fbf2-4e89-afbc-543ce47e0615/volume-data-source-validator/0.log"
Sep 16 13:23:58.510195 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:58.510155    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-mq764" event={"ID":"a75ad2bd-3ed0-4831-ab47-94d160e3b6dc","Type":"ContainerStarted","Data":"7460600da300c1ef485513abce10ec6f37d711ee1dcea4f7c5be342de2ef025c"}
Sep 16 13:23:58.510384 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:58.510222    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-mq764" event={"ID":"a75ad2bd-3ed0-4831-ab47-94d160e3b6dc","Type":"ContainerStarted","Data":"4bde8a244da66c5ee41861003c9c785df7dbc8d1b58457964bef203d7390fad4"}
Sep 16 13:23:58.511078 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:58.511053    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-mq764"
Sep 16 13:23:58.545324 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:58.545274    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-mq764" podStartSLOduration=2.5452493499999997 podStartE2EDuration="2.54524935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:23:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:23:58.541616369 +0000 UTC m=+2744.988849011" watchObservedRunningTime="2026-09-16 13:23:58.54524935 +0000 UTC m=+2744.992481991"
Sep 16 13:23:59.564745 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:59.564711    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ct9vp_4f5ad86c-a72b-4f7e-aa27-484b2e8a3261/dns/0.log"
Sep 16 13:23:59.598310 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:59.598282    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ct9vp_4f5ad86c-a72b-4f7e-aa27-484b2e8a3261/kube-rbac-proxy/0.log"
Sep 16 13:23:59.840951 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:23:59.840867    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t47mb_c44032b0-4e54-42d9-945f-7d401e20d50c/dns-node-resolver/0.log"
Sep 16 13:24:00.574668 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:24:00.574610    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8bxln_d3f21250-f182-428a-a11e-99f2a158ca61/node-ca/0.log"
Sep 16 13:24:02.024530 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:24:02.024506    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s_ef53d612-0d2e-426f-a926-bcee1b7d4827/istio-proxy/0.log"
Sep 16 13:24:02.566213 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:24:02.566148    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9rfz6_be45e9ae-2a43-4cd9-bd86-541bc2a97291/istio-proxy/0.log"
Sep 16 13:24:02.571384 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:24:02.571356    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9rfz6_be45e9ae-2a43-4cd9-bd86-541bc2a97291/istio-proxy/1.log"
Sep 16 13:24:02.713744 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:24:02.713708    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-944d5c59d-l5g2b_b04ec1dc-5e71-4fed-abc5-6875befad478/router/0.log"
Sep 16 13:24:03.821735 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:24:03.821704    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pjjgz_6bd5627d-472c-43bf-a550-8f30201dee34/serve-healthcheck-canary/0.log"
Sep 16 13:24:04.754268 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:24:04.754234    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nps4n_fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7/kube-rbac-proxy/0.log"
Sep 16 13:24:04.786842 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:24:04.786814    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nps4n_fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7/exporter/0.log"
Sep 16 13:24:04.822210 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:24:04.822167    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nps4n_fcd51b6e-cc62-4d4d-9ff0-d2fce50916c7/extractor/0.log"
Sep 16 13:24:05.533942 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:24:05.533906    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-mq764"
Sep 16 13:24:07.909250 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:24:07.909210    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-ssfwx_9ed52a74-3e4e-4519-b070-f1bca5653647/manager/0.log"
Sep 16 13:24:07.945473 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:24:07.945434    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s_ef53d612-0d2e-426f-a926-bcee1b7d4827/istio-proxy/0.log"
Sep 16 13:24:08.374068 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:24:08.374013    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c7bc95455-h5zdt_ad54ab6a-787b-471a-a648-1badf8c1b94c/manager/0.log"
Sep 16 13:24:08.421659 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:24:08.421627    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c7bc95455-h5zdt_ad54ab6a-787b-471a-a648-1badf8c1b94c/copy-manifests/0.log"
Sep 16 13:24:08.492181 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:24:08.492153    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9rfz6_be45e9ae-2a43-4cd9-bd86-541bc2a97291/istio-proxy/0.log"
Sep 16 13:24:08.501377 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:24:08.501352    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9rfz6_be45e9ae-2a43-4cd9-bd86-541bc2a97291/istio-proxy/1.log"
Sep 16 13:24:08.583967 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:24:08.583937    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-944d5c59d-l5g2b_b04ec1dc-5e71-4fed-abc5-6875befad478/router/0.log"
Sep 16 13:24:09.942142 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:24:09.942045    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-575fbb86f4-qr6g8_1c9eb370-b426-4f43-ad86-7be180866403/manager/0.log"
Sep 16 13:24:10.343828 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:24:10.343793    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-x225s_ef53d612-0d2e-426f-a926-bcee1b7d4827/istio-proxy/0.log"
Sep 16 13:24:10.825084 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:24:10.825044    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9rfz6_be45e9ae-2a43-4cd9-bd86-541bc2a97291/istio-proxy/0.log"
Sep 16 13:24:10.829348 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:24:10.829315    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9rfz6_be45e9ae-2a43-4cd9-bd86-541bc2a97291/istio-proxy/1.log"
Sep 16 13:24:10.922981 ip-10-0-133-190 kubenswrapper[2577]: I0916 13:24:10.922940    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-944d5c59d-l5g2b_b04ec1dc-5e71-4fed-abc5-6875befad478/router/0.log"