Sep 11 17:18:39.491292 ip-10-0-133-252 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 17:18:39.886358 ip-10-0-133-252 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 17:18:39.886358 ip-10-0-133-252 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 17:18:39.886358 ip-10-0-133-252 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 17:18:39.886358 ip-10-0-133-252 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 17:18:39.886358 ip-10-0-133-252 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 17:18:39.889277 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.889190    2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 17:18:39.893713 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893698    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:18:39.893713 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893713    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:18:39.893780 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893717    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:18:39.893780 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893722    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:18:39.893780 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893725    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:18:39.893780 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893729    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:18:39.893780 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893733    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:18:39.893780 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893737    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:18:39.893780 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893742    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:18:39.893780 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893745    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:18:39.893780 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893748    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:18:39.893780 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893751    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:18:39.893780 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893755    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:18:39.893780 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893758    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:18:39.893780 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893761    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:18:39.893780 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893764    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:18:39.893780 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893768    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:18:39.893780 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893770    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:18:39.893780 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893774    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:18:39.893780 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893777    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:18:39.893780 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893780    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:18:39.894255 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893783    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:18:39.894255 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893785    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:18:39.894255 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893788    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:18:39.894255 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893791    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:18:39.894255 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893794    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:18:39.894255 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893796    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:18:39.894255 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893799    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:18:39.894255 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893802    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:18:39.894255 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893804    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:18:39.894255 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893807    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:18:39.894255 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893809    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:18:39.894255 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893812    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:18:39.894255 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893814    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:18:39.894255 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893817    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:18:39.894255 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893819    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:18:39.894255 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893822    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:18:39.894255 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893825    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:18:39.894255 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893827    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:18:39.894255 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893831    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:18:39.894255 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893833    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:18:39.894773 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893836    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:18:39.894773 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893838    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:18:39.894773 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893841    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:18:39.894773 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893843    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:18:39.894773 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893845    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:18:39.894773 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893848    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:18:39.894773 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893850    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:18:39.894773 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893853    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:18:39.894773 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893855    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:18:39.894773 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893858    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:18:39.894773 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893860    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:18:39.894773 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893863    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:18:39.894773 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893866    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:18:39.894773 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893869    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:18:39.894773 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893872    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:18:39.894773 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893875    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:18:39.894773 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893878    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:18:39.894773 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893880    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:18:39.894773 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893883    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:18:39.894773 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893885    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:18:39.895306 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893888    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:18:39.895306 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893891    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:18:39.895306 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893893    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:18:39.895306 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893896    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:18:39.895306 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893898    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:18:39.895306 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893902    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:18:39.895306 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893906    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:18:39.895306 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893909    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:18:39.895306 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893912    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:18:39.895306 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893914    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:18:39.895306 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893917    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:18:39.895306 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893921    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:18:39.895306 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893923    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:18:39.895306 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893926    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:18:39.895306 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893928    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:18:39.895306 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893931    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:18:39.895306 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893933    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:18:39.895306 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893936    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:18:39.895306 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893938    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:18:39.895781 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893942    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:18:39.895781 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893944    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:18:39.895781 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893947    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:18:39.895781 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893950    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:18:39.895781 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893952    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:18:39.895781 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893955    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:18:39.895781 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893959    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:18:39.895781 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893962    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:18:39.895781 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893965    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:18:39.895781 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893967    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:18:39.895781 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893970    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:18:39.895781 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893973    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:18:39.895781 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893975    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:18:39.895781 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893978    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:18:39.895781 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893981    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:18:39.895781 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893984    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:18:39.895781 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.893987    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:18:39.895781 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894002    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:18:39.895781 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894004    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:18:39.895781 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894716    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:18:39.896543 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894722    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:18:39.896543 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894726    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:18:39.896543 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894729    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:18:39.896543 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894732    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:18:39.896543 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894735    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:18:39.896543 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894740    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:18:39.896543 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894743    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:18:39.896543 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894746    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:18:39.896543 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894749    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:18:39.896543 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894751    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:18:39.896543 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894754    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:18:39.896543 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894757    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:18:39.896543 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894759    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:18:39.896543 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894762    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:18:39.896543 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894764    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:18:39.896543 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894767    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:18:39.896543 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894770    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:18:39.896543 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894773    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:18:39.896543 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894776    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:18:39.896543 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894779    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:18:39.897202 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894782    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:18:39.897202 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894785    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:18:39.897202 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894788    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:18:39.897202 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894790    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:18:39.897202 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894793    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:18:39.897202 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894796    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:18:39.897202 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894798    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:18:39.897202 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894801    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:18:39.897202 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894803    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:18:39.897202 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894806    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:18:39.897202 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894808    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:18:39.897202 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894811    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:18:39.897202 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894813    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:18:39.897202 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894816    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:18:39.897202 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894818    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:18:39.897202 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894821    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:18:39.897202 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894823    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:18:39.897202 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894826    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:18:39.897202 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894830    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:18:39.897202 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894833    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:18:39.897778 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894835    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:18:39.897778 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894838    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:18:39.897778 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894840    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:18:39.897778 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894844    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:18:39.897778 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894848    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:18:39.897778 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894851    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:18:39.897778 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894854    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:18:39.897778 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894857    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:18:39.897778 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894859    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:18:39.897778 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894863    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:18:39.897778 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894866    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:18:39.897778 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894869    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:18:39.897778 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894872    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:18:39.897778 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894874    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:18:39.897778 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894877    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:18:39.897778 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894880    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:18:39.897778 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894883    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:18:39.897778 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894885    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:18:39.897778 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894888    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:18:39.898317 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894890    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:18:39.898317 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894893    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:18:39.898317 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894895    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:18:39.898317 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894898    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:18:39.898317 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894900    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:18:39.898317 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894905    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:18:39.898317 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894909    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:18:39.898317 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894911    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:18:39.898317 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894914    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:18:39.898317 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894916    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:18:39.898317 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894919    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:18:39.898317 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894921    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:18:39.898317 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894926    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:18:39.898317 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894929    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:18:39.898317 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894932    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:18:39.898317 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894934    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:18:39.898317 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894937    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:18:39.898317 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894939    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:18:39.898317 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894942    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:18:39.898784 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894945    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:18:39.898784 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894947    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:18:39.898784 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894950    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:18:39.898784 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894953    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:18:39.898784 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894956    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:18:39.898784 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894958    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:18:39.898784 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894961    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:18:39.898784 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894963    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:18:39.898784 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894966    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:18:39.898784 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894968    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:18:39.898784 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894971    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:18:39.898784 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894974    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:18:39.898784 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894976    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:18:39.898784 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894979    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:18:39.898784 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894982    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:18:39.898784 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894984    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:18:39.898784 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.894986    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:18:39.898784 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895004    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:18:39.898784 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895008    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:18:39.899286 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895012    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:18:39.899286 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895378    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:18:39.899286 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895383    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:18:39.899286 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895386    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:18:39.899286 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895389    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:18:39.899286 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895391    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:18:39.899286 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895401    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:18:39.899286 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895404    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:18:39.899286 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895407    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:18:39.899286 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895410    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:18:39.899286 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895413    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:18:39.899286 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895416    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:18:39.899286 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895421    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:18:39.899286 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895424    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:18:39.899286 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895427    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:18:39.899286 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895430    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:18:39.899286 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895433    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:18:39.899286 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895437    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:18:39.899286 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895440    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:18:39.899742 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895443    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:18:39.899742 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895445    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:18:39.899742 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895448    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:18:39.899742 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895451    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:18:39.899742 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895453    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:18:39.899742 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895456    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:18:39.899742 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895459    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:18:39.899742 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895462    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:18:39.899742 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895465    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:18:39.899742 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895467    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:18:39.899742 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895470    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:18:39.899742 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895472    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:18:39.899742 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895475    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:18:39.899742 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895477    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:18:39.899742 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895480    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:18:39.899742 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895482    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:18:39.899742 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895485    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:18:39.899742 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895487    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:18:39.899742 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895490    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:18:39.899742 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895492    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:18:39.900250 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895495    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:18:39.900250 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895498    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:18:39.900250 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895500    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:18:39.900250 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895503    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:18:39.900250 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895505    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:18:39.900250 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895508    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:18:39.900250 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895510    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:18:39.900250 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895513    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:18:39.900250 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895516    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:18:39.900250 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895518    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:18:39.900250 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895522    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:18:39.900250 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895524    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:18:39.900250 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895527    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:18:39.900250 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895529    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:18:39.900250 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895532    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:18:39.900250 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895534    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:18:39.900250 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895537    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:18:39.900250 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895539    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:18:39.900250 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895542    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:18:39.900250 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895545    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:18:39.900736 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895547    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:18:39.900736 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895550    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:18:39.900736 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895554    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:18:39.900736 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895557    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:18:39.900736 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895560    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:18:39.900736 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895563    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:18:39.900736 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895566    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:18:39.900736 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895568    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:18:39.900736 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895571    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:18:39.900736 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895574    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:18:39.900736 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895577    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:18:39.900736 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895579    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:18:39.900736 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895582    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:18:39.900736 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895585    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:18:39.900736 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895587    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:18:39.900736 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895589    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:18:39.900736 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895592    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:18:39.900736 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895594    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:18:39.900736 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895597    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:18:39.901211 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895600    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:18:39.901211 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895602    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:18:39.901211 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895605    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:18:39.901211 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895608    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:18:39.901211 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895610    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:18:39.901211 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895613    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:18:39.901211 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895615    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:18:39.901211 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895618    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:18:39.901211 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895621    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:18:39.901211 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895623    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:18:39.901211 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895626    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:18:39.901211 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895628    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:18:39.901211 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895631    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:18:39.901211 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895633    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:18:39.901211 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895636    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:18:39.901211 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895638    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:18:39.901211 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895642    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:18:39.901211 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895648    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:18:39.901211 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895651    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:18:39.901211 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895653    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:18:39.901691 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895656    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:18:39.901691 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.895658    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:18:39.901691 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897202    2562 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 17:18:39.901691 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897254    2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 17:18:39.901691 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897270    2562 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 17:18:39.901691 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897275    2562 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 17:18:39.901691 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897280    2562 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 17:18:39.901691 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897283    2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 17:18:39.901691 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897288    2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 17:18:39.901691 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897293    2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 17:18:39.901691 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897296    2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 17:18:39.901691 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897299    2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 17:18:39.901691 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897302    2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 17:18:39.901691 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897306    2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 17:18:39.901691 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897309    2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 17:18:39.901691 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897312    2562 flags.go:64] FLAG: --cgroup-root=""
Sep 11 17:18:39.901691 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897316    2562 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 17:18:39.901691 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897321    2562 flags.go:64] FLAG: --client-ca-file=""
Sep 11 17:18:39.901691 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897325    2562 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 17:18:39.901691 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897330    2562 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 17:18:39.901691 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897336    2562 flags.go:64] FLAG: --cluster-domain=""
Sep 11 17:18:39.901691 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897341    2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 17:18:39.901691 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897345    2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 17:18:39.902263 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897348    2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 17:18:39.902263 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897351    2562 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 17:18:39.902263 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897355    2562 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 17:18:39.902263 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897358    2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 17:18:39.902263 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897362    2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 17:18:39.902263 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897365    2562 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 17:18:39.902263 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897368    2562 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 17:18:39.902263 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897371    2562 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 17:18:39.902263 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897374    2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 17:18:39.902263 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897379    2562 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 17:18:39.902263 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897382    2562 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 17:18:39.902263 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897391    2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 17:18:39.902263 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897394    2562 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 17:18:39.902263 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897397    2562 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 17:18:39.902263 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897400    2562 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 17:18:39.902263 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897404    2562 flags.go:64] FLAG: --enable-server="true"
Sep 11 17:18:39.902263 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897408    2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 17:18:39.902263 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897417    2562 flags.go:64] FLAG: --event-burst="100"
Sep 11 17:18:39.902263 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897422    2562 flags.go:64] FLAG: --event-qps="50"
Sep 11 17:18:39.902263 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897426    2562 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 17:18:39.902263 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897429    2562 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 17:18:39.902263 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897432    2562 flags.go:64] FLAG: --eviction-hard=""
Sep 11 17:18:39.902263 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897436    2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 17:18:39.902263 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897439    2562 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 17:18:39.902835 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897442    2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 17:18:39.902835 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897445    2562 flags.go:64] FLAG: --eviction-soft=""
Sep 11 17:18:39.902835 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897448    2562 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 17:18:39.902835 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897451    2562 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 17:18:39.902835 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897454    2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 17:18:39.902835 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897457    2562 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 17:18:39.902835 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897460    2562 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 17:18:39.902835 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897462    2562 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 17:18:39.902835 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897465    2562 flags.go:64] FLAG: --feature-gates=""
Sep 11 17:18:39.902835 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897469    2562 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 17:18:39.902835 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897473    2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 17:18:39.902835 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897476    2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 17:18:39.902835 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897479    2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 17:18:39.902835 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897482    2562 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 17:18:39.902835 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897485    2562 flags.go:64] FLAG: --help="false"
Sep 11 17:18:39.902835 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897490    2562 flags.go:64] FLAG: --hostname-override="ip-10-0-133-252.ec2.internal"
Sep 11 17:18:39.902835 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897496    2562 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 17:18:39.902835 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897502    2562 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 17:18:39.902835 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897505    2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 17:18:39.902835 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897509    2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 17:18:39.902835 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897514    2562 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 17:18:39.902835 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897517    2562 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 17:18:39.902835 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897520    2562 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 17:18:39.903442 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897524    2562 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 17:18:39.903442 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897527    2562 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 17:18:39.903442 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897529    2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 17:18:39.903442 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897533    2562 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 17:18:39.903442 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897536    2562 flags.go:64] FLAG: --kube-reserved=""
Sep 11 17:18:39.903442 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897539    2562 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 17:18:39.903442 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897541    2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 17:18:39.903442 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897544    2562 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 17:18:39.903442 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897547    2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 17:18:39.903442 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897550    2562 flags.go:64] FLAG: --lock-file=""
Sep 11 17:18:39.903442 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897553    2562 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 17:18:39.903442 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897556    2562 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 17:18:39.903442 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897559    2562 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 17:18:39.903442 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897564    2562 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 17:18:39.903442 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897568    2562 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 17:18:39.903442 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897574    2562 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 17:18:39.903442 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897579    2562 flags.go:64] FLAG: --logging-format="text"
Sep 11 17:18:39.903442 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897583    2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 17:18:39.903442 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897587    2562 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 17:18:39.903442 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897590    2562 flags.go:64] FLAG: --manifest-url=""
Sep 11 17:18:39.903442 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897593    2562 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 17:18:39.903442 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897599    2562 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 17:18:39.903442 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897602    2562 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 17:18:39.903442 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897606    2562 flags.go:64] FLAG: --max-pods="110"
Sep 11 17:18:39.903442 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897609    2562 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 17:18:39.904117 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897612    2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 17:18:39.904117 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897615    2562 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 17:18:39.904117 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897618    2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 17:18:39.904117 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897622    2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 17:18:39.904117 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897628    2562 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 17:18:39.904117 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897631    2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 17:18:39.904117 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897639    2562 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 17:18:39.904117 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897642    2562 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 17:18:39.904117 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897646    2562 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 17:18:39.904117 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897651    2562 flags.go:64] FLAG: --pod-cidr=""
Sep 11 17:18:39.904117 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897656    2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 17:18:39.904117 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897664    2562 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 17:18:39.904117 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897667    2562 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 17:18:39.904117 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897670    2562 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 17:18:39.904117 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897673    2562 flags.go:64] FLAG: --port="10250"
Sep 11 17:18:39.904117 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897676    2562 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 17:18:39.904117 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897679    2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-074d1fb7327648629"
Sep 11 17:18:39.904117 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897683    2562 flags.go:64] FLAG: --qos-reserved=""
Sep 11 17:18:39.904117 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897685    2562 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 17:18:39.904117 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897688    2562 flags.go:64] FLAG: --register-node="true"
Sep 11 17:18:39.904117 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897691    2562 flags.go:64] FLAG: --register-with-taints=""
Sep 11 17:18:39.904117 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897695    2562 flags.go:64] FLAG: --registry-burst="10"
Sep 11 17:18:39.904117 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897698    2562 flags.go:64] FLAG: --registry-qps="5"
Sep 11 17:18:39.904117 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897702    2562 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 17:18:39.904708 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897705    2562 flags.go:64] FLAG: --reserved-memory=""
Sep 11 17:18:39.904708 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897709    2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 17:18:39.904708 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897712    2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 17:18:39.904708 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897715    2562 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 17:18:39.904708 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897719    2562 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 17:18:39.904708 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897722    2562 flags.go:64] FLAG: --runonce="false"
Sep 11 17:18:39.904708 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897725    2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 17:18:39.904708 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897729    2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 17:18:39.904708 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897734    2562 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 17:18:39.904708 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897739    2562 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 17:18:39.904708 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897743    2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 17:18:39.904708 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897747    2562 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 17:18:39.904708 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897750    2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 17:18:39.904708 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897755    2562 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 17:18:39.904708 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897758    2562 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 17:18:39.904708 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897760    2562 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 17:18:39.904708 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897763    2562 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 17:18:39.904708 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897766    2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 17:18:39.904708 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897769    2562 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 17:18:39.904708 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897772    2562 flags.go:64] FLAG: --system-cgroups=""
Sep 11 17:18:39.904708 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897775    2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 17:18:39.904708 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897782    2562 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 17:18:39.904708 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897785    2562 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 17:18:39.904708 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897788    2562 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 17:18:39.904708 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897792    2562 flags.go:64] FLAG: --tls-min-version=""
Sep 11 17:18:39.905383 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897795    2562 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 17:18:39.905383 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897798    2562 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 17:18:39.905383 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897801    2562 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 17:18:39.905383 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897804    2562 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 17:18:39.905383 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897807    2562 flags.go:64] FLAG: --v="2"
Sep 11 17:18:39.905383 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897813    2562 flags.go:64] FLAG: --version="false"
Sep 11 17:18:39.905383 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897820    2562 flags.go:64] FLAG: --vmodule=""
Sep 11 17:18:39.905383 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897826    2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 17:18:39.905383 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.897830    2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 17:18:39.905383 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.897956    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:18:39.905383 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.897963    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:18:39.905383 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.897968    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:18:39.905383 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.897972    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:18:39.905383 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.897977    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:18:39.905383 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.897982    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:18:39.905383 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.897986    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:18:39.905383 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898005    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:18:39.905383 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898009    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:18:39.905383 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898011    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:18:39.905383 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898014    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:18:39.905383 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898018    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:18:39.905902 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898021    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:18:39.905902 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898025    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:18:39.905902 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898027    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:18:39.905902 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898030    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:18:39.905902 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898033    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:18:39.905902 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898035    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:18:39.905902 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898038    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:18:39.905902 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898041    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:18:39.905902 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898043    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:18:39.905902 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898046    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:18:39.905902 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898049    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:18:39.905902 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898052    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:18:39.905902 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898056    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:18:39.905902 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898061    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:18:39.905902 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898065    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:18:39.905902 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898069    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:18:39.905902 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898072    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:18:39.905902 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898074    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:18:39.905902 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898077    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:18:39.906390 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898081    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:18:39.906390 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898084    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:18:39.906390 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898086    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:18:39.906390 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898090    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:18:39.906390 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898092    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:18:39.906390 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898095    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:18:39.906390 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898097    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:18:39.906390 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898100    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:18:39.906390 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898102    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:18:39.906390 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898105    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:18:39.906390 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898107    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:18:39.906390 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898110    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:18:39.906390 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898114    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:18:39.906390 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898116    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:18:39.906390 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898119    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:18:39.906390 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898122    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:18:39.906390 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898124    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:18:39.906390 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898127    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:18:39.906390 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898132    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:18:39.906853 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898137    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:18:39.906853 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898142    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:18:39.906853 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898147    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:18:39.906853 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898151    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:18:39.906853 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898155    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:18:39.906853 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898157    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:18:39.906853 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898160    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:18:39.906853 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898163    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:18:39.906853 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898166    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:18:39.906853 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898168    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:18:39.906853 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898171    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:18:39.906853 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898173    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:18:39.906853 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898176    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:18:39.906853 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898180    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:18:39.906853 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898184    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:18:39.906853 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898187    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:18:39.906853 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898190    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:18:39.906853 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898192    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:18:39.906853 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898195    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:18:39.906853 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898197    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:18:39.907408 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898200    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:18:39.907408 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898203    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:18:39.907408 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898205    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:18:39.907408 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898208    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:18:39.907408 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898210    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:18:39.907408 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898215    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:18:39.907408 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898219    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:18:39.907408 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898223    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:18:39.907408 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898227    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:18:39.907408 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898231    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:18:39.907408 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898235    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:18:39.907408 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898237    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:18:39.907408 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898240    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:18:39.907408 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898243    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:18:39.907408 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898245    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:18:39.907408 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898248    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:18:39.907408 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898250    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:18:39.907408 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898253    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:18:39.907408 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898256    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:18:39.907408 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898258    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:18:39.907903 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898261    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:18:39.907903 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898263    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:18:39.907903 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898266    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:18:39.907903 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898269    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:18:39.907903 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898271    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:18:39.907903 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898274    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:18:39.907903 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898277    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:18:39.907903 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898280    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:18:39.907903 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.898283    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:18:39.907903 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.899066    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 17:18:39.907903 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.906107    2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 17:18:39.907903 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.906122    2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 17:18:39.907903 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.906854    2562 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 17:18:39.907903 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.906860    2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 17:18:39.907903 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.906918    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:18:39.908342 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.906923    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:18:39.908342 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.906926    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:18:39.908342 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.906930    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:18:39.908342 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.906933    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:18:39.908342 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.906936    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:18:39.908342 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.906939    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:18:39.908342 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.906942    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:18:39.908342 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.906945    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:18:39.908342 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.906948    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:18:39.908342 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.906951    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:18:39.908342 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.906954    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:18:39.908342 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.906957    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:18:39.908342 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.906960    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:18:39.908342 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.906963    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:18:39.908342 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.906966    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:18:39.908342 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.906969    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:18:39.908342 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.906972    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:18:39.908342 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.906974    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:18:39.908342 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.906978    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:18:39.908826 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.906982    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:18:39.908826 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.906985    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:18:39.908826 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907002    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:18:39.908826 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907006    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:18:39.908826 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907009    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:18:39.908826 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907013    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:18:39.908826 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907017    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:18:39.908826 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907020    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:18:39.908826 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907023    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:18:39.908826 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907026    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:18:39.908826 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907029    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:18:39.908826 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907032    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:18:39.908826 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907037    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:18:39.908826 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907039    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:18:39.908826 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907042    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:18:39.908826 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907046    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:18:39.908826 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907049    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:18:39.908826 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907052    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:18:39.908826 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907055    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:18:39.908826 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907057    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:18:39.909327 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907060    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:18:39.909327 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907062    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:18:39.909327 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907065    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:18:39.909327 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907068    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:18:39.909327 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907071    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:18:39.909327 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907073    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:18:39.909327 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907076    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:18:39.909327 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907078    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:18:39.909327 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907081    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:18:39.909327 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907083    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:18:39.909327 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907086    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:18:39.909327 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907088    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:18:39.909327 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907091    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:18:39.909327 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907094    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:18:39.909327 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907096    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:18:39.909327 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907099    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:18:39.909327 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907101    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:18:39.909327 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907104    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:18:39.909327 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907108    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:18:39.909327 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907110    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:18:39.909811 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907113    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:18:39.909811 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907117    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:18:39.909811 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907120    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:18:39.909811 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907124    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:18:39.909811 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907127    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:18:39.909811 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907130    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:18:39.909811 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907133    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:18:39.909811 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907135    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:18:39.909811 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907138    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:18:39.909811 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907140    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:18:39.909811 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907143    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:18:39.909811 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907146    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:18:39.909811 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907148    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:18:39.909811 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907151    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:18:39.909811 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907153    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:18:39.909811 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907156    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:18:39.909811 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907158    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:18:39.909811 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907161    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:18:39.909811 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907163    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:18:39.910285 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907166    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:18:39.910285 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907169    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:18:39.910285 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907171    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:18:39.910285 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907174    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:18:39.910285 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907176    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:18:39.910285 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907179    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:18:39.910285 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907181    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:18:39.910285 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907184    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:18:39.910285 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907186    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:18:39.910285 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907189    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:18:39.910285 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907191    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:18:39.910285 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907194    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:18:39.910285 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907198    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:18:39.910285 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907200    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:18:39.910285 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907203    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:18:39.910285 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907206    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:18:39.910285 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907208    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:18:39.910285 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907211    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:18:39.910285 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907213    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:18:39.910777 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907216    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:18:39.910777 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.907221    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 17:18:39.910777 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907333    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:18:39.910777 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907338    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:18:39.910777 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907341    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:18:39.910777 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907343    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:18:39.910777 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907346    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:18:39.910777 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907349    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:18:39.910777 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907352    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:18:39.910777 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907354    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:18:39.910777 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907357    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:18:39.910777 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907359    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:18:39.910777 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907362    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:18:39.910777 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907364    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:18:39.910777 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907367    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:18:39.911170 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907369    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:18:39.911170 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907372    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:18:39.911170 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907375    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:18:39.911170 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907379    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:18:39.911170 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907382    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:18:39.911170 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907384    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:18:39.911170 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907387    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:18:39.911170 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907389    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:18:39.911170 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907392    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:18:39.911170 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907394    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:18:39.911170 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907397    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:18:39.911170 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907400    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:18:39.911170 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907403    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:18:39.911170 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907405    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:18:39.911170 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907408    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:18:39.911170 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907411    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:18:39.911170 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907414    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:18:39.911170 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907417    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:18:39.911170 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907419    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:18:39.911629 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907422    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:18:39.911629 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907424    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:18:39.911629 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907427    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:18:39.911629 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907429    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:18:39.911629 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907432    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:18:39.911629 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907435    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:18:39.911629 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907437    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:18:39.911629 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907440    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:18:39.911629 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907442    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:18:39.911629 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907445    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:18:39.911629 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907447    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:18:39.911629 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907450    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:18:39.911629 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907452    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:18:39.911629 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907455    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:18:39.911629 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907457    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:18:39.911629 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907460    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:18:39.911629 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907462    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:18:39.911629 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907465    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:18:39.911629 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907467    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:18:39.911629 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907470    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:18:39.912156 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907472    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:18:39.912156 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907475    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:18:39.912156 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907477    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:18:39.912156 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907480    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:18:39.912156 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907483    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:18:39.912156 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907486    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:18:39.912156 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907489    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:18:39.912156 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907491    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:18:39.912156 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907494    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:18:39.912156 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907496    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:18:39.912156 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907499    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:18:39.912156 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907502    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:18:39.912156 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907504    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:18:39.912156 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907508    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:18:39.912156 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907511    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:18:39.912156 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907514    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:18:39.912156 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907517    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:18:39.912156 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907519    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:18:39.912156 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907521    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:18:39.912624 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907524    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:18:39.912624 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907527    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:18:39.912624 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907529    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:18:39.912624 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907532    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:18:39.912624 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907534    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:18:39.912624 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907537    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:18:39.912624 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907539    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:18:39.912624 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907542    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:18:39.912624 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907544    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:18:39.912624 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907547    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:18:39.912624 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907549    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:18:39.912624 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907552    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:18:39.912624 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907554    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:18:39.912624 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907557    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:18:39.912624 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907561    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:18:39.912624 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907564    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:18:39.912624 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907567    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:18:39.912624 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907570    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:18:39.912624 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907573    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:18:39.912624 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907582    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:18:39.913112 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907585    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:18:39.913112 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907588    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:18:39.913112 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907591    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:18:39.913112 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907594    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:18:39.913112 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907597    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:18:39.913112 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907600    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:18:39.913112 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907602    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:18:39.913112 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:39.907605    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:18:39.913112 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.907609    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 17:18:39.913112 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.907706    2562 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 17:18:39.913112 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.909740    2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 17:18:39.913112 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.910649    2562 server.go:1019] "Starting client certificate rotation"
Sep 11 17:18:39.913112 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.910759    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 17:18:39.913112 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.910796    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 17:18:39.913466 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.913300    2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 17:18:39.916848 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.916830    2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 17:18:39.930220 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.930198    2562 log.go:25] "Validated CRI v1 runtime API"
Sep 11 17:18:39.934213 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.934193    2562 log.go:25] "Validated CRI v1 image API"
Sep 11 17:18:39.935067 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.935049    2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 17:18:39.938549 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.938532    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 17:18:39.939603 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.939585    2562 fs.go:135] Filesystem UUIDs: map[3ee2539f-c4fc-4d21-abc0-c3ecd6a986d9:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 841558b0-85cc-4ff9-beb4-d9851af53110:/dev/nvme0n1p3]
Sep 11 17:18:39.939644 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.939603    2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 17:18:39.945661 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.945553    2562 manager.go:216] Machine: {Timestamp:2026-09-11 17:18:39.943262803 +0000 UTC m=+0.337180249 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101165 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ca2daff4555be87fe2b7f84654680 SystemUUID:ec2ca2da-ff45-55be-87fe-2b7f84654680 BootID:e4fd6362-142d-4ca4-bd1b-1df2208b1ad6 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:7f:55:fd:f5:b3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:7f:55:fd:f5:b3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3e:db:30:94:81:91 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 17:18:39.945661 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.945659    2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 17:18:39.945765 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.945745    2562 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 17:18:39.949277 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.949251    2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 17:18:39.949437 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.949279    2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-252.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 17:18:39.950112 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.950102    2562 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 17:18:39.950150 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.950114    2562 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 17:18:39.950150 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.950121    2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 17:18:39.950723 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.950714    2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 17:18:39.951552 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.951543    2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 17:18:39.955347 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.955336    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 17:18:39.955463 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.955454    2562 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 17:18:39.955492 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.955486    2562 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 17:18:39.955520 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.955499    2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 17:18:39.955520 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.955513    2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 17:18:39.955584 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.955522    2562 kubelet.go:398] "Adding apiserver pod source"
Sep 11 17:18:39.955584 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.955531    2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 17:18:39.955946 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.955932    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 17:18:39.956027 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.955955    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 17:18:39.958356 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.958336    2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 17:18:39.960629 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.960615    2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 17:18:39.960684 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.960646    2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 17:18:39.962197 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.962184    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 17:18:39.962248 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.962203    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 17:18:39.962248 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.962211    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 17:18:39.962248 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.962220    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 17:18:39.962248 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.962230    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 17:18:39.962248 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.962236    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 17:18:39.962248 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.962249    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 17:18:39.962403 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.962254    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 17:18:39.962403 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.962261    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 17:18:39.962403 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.962267    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 17:18:39.962403 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.962280    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 17:18:39.962403 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.962285    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 17:18:39.963598 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.963586    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 17:18:39.963647 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.963600    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 17:18:39.965587 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:39.965550    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-252.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 17:18:39.965587 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:39.965544    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 17:18:39.965674 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.965627    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8t4nf"
Sep 11 17:18:39.965674 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.965620    2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-252.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 17:18:39.967355 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.967342    2562 server.go:1268] "Started kubelet"
Sep 11 17:18:39.967558 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.967472    2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 17:18:39.967631 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.967611    2562 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 17:18:39.968159 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.968112    2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 17:18:39.968555 ip-10-0-133-252 systemd[1]: Started Kubernetes Kubelet.
Sep 11 17:18:39.969866 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.969842    2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 17:18:39.970851 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.970832    2562 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 17:18:39.971780 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.971295    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8t4nf"
Sep 11 17:18:39.975522 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.975443    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 17:18:39.975983 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.975962    2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 17:18:39.976752 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:39.976717    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-252.ec2.internal\" not found"
Sep 11 17:18:39.977439 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.977418    2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 17:18:39.978117 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.978099    2562 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 17:18:39.978217 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.978199    2562 factory.go:153] Registering CRI-O factory
Sep 11 17:18:39.978273 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.978222    2562 factory.go:223] Registration of the crio container factory successfully
Sep 11 17:18:39.978318 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.978281    2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 17:18:39.978318 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.978291    2562 factory.go:55] Registering systemd factory
Sep 11 17:18:39.978318 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.978308    2562 factory.go:223] Registration of the systemd container factory successfully
Sep 11 17:18:39.978443 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.978325    2562 factory.go:103] Registering Raw factory
Sep 11 17:18:39.978443 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.978342    2562 manager.go:1196] Started watching for new ooms in manager
Sep 11 17:18:39.978911 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.978894    2562 manager.go:349] Starting recovery of all containers
Sep 11 17:18:39.979157 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.979142    2562 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 17:18:39.979230 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:39.979189    2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 17:18:39.979345 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.979326    2562 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 17:18:39.979345 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.979338    2562 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 17:18:39.984579 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.984561    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:18:39.988551 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.988502    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 17:18:39.988741 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:39.988715    2562 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-133-252.ec2.internal\" not found" node="ip-10-0-133-252.ec2.internal"
Sep 11 17:18:39.990914 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.990733    2562 manager.go:354] Recovery completed
Sep 11 17:18:39.995687 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.995670    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:18:39.998027 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.998007    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-252.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:18:39.998127 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.998040    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-252.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:18:39.998127 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.998050    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-252.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:18:39.999168 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.999152    2562 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 17:18:39.999168 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.999166    2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 17:18:39.999273 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.999180    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:18:39.999273 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.999185    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 17:18:39.999973 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.999957    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-252.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:18:40.000086 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:39.999983    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-252.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:18:40.000086 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.000015    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-252.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:18:40.000386 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.000373    2562 policy_none.go:49] "None policy: Start"
Sep 11 17:18:40.000420 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.000390    2562 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 17:18:40.000420 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.000400    2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 17:18:40.001390 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.001379    2562 policy_none.go:47] "Start"
Sep 11 17:18:40.030831 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.030815    2562 manager.go:337] "Starting Device Plugin manager"
Sep 11 17:18:40.043338 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:40.030879    2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 17:18:40.043338 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.030894    2562 server.go:87] "Starting device plugin registration server"
Sep 11 17:18:40.043338 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.031239    2562 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 17:18:40.043338 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.031270    2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 17:18:40.043338 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.031365    2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 17:18:40.043338 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.031444    2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 17:18:40.043338 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.031452    2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 17:18:40.043338 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:40.031936    2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 17:18:40.043338 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:40.031980    2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-252.ec2.internal\" not found"
Sep 11 17:18:40.118810 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.118782    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 17:18:40.118978 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.118850    2562 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 17:18:40.118978 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.118879    2562 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 17:18:40.119092 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:40.118988    2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 17:18:40.121490 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.121466    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:18:40.131600 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.131585    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:18:40.132475 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.132458    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-252.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:18:40.132533 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.132494    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-252.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:18:40.132533 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.132507    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-252.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:18:40.132533 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.132529    2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-252.ec2.internal"
Sep 11 17:18:40.140720 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.140679    2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-252.ec2.internal"
Sep 11 17:18:40.140720 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:40.140700    2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-252.ec2.internal\": node \"ip-10-0-133-252.ec2.internal\" not found"
Sep 11 17:18:40.156782 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:40.156763    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-252.ec2.internal\" not found"
Sep 11 17:18:40.220115 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.220040    2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-252.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-252.ec2.internal"]
Sep 11 17:18:40.220289 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.220156    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:18:40.221243 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.221226    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-252.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:18:40.221371 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.221253    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-252.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:18:40.221371 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.221263    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-252.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:18:40.222507 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.222495    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:18:40.222857 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.222623    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-252.ec2.internal"
Sep 11 17:18:40.222927 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.222916    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:18:40.224887 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.224866    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-252.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:18:40.225022 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.224901    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-252.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:18:40.225022 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.224915    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-252.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:18:40.225022 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.224924    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-252.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:18:40.225022 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.224947    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-252.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:18:40.225022 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.224957    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-252.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:18:40.226040 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.226027    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-252.ec2.internal"
Sep 11 17:18:40.226098 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.226052    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:18:40.226726 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.226708    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-252.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:18:40.226827 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.226733    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-252.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:18:40.226827 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.226742    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-252.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:18:40.245521 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:40.245497    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-252.ec2.internal\" not found" node="ip-10-0-133-252.ec2.internal"
Sep 11 17:18:40.249613 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:40.249596    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-252.ec2.internal\" not found" node="ip-10-0-133-252.ec2.internal"
Sep 11 17:18:40.256909 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:40.256896    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-252.ec2.internal\" not found"
Sep 11 17:18:40.281760 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.281739    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4d9878da489eb3fb19c741850664c940-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-252.ec2.internal\" (UID: \"4d9878da489eb3fb19c741850664c940\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-252.ec2.internal"
Sep 11 17:18:40.281901 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.281764    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d9878da489eb3fb19c741850664c940-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-252.ec2.internal\" (UID: \"4d9878da489eb3fb19c741850664c940\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-252.ec2.internal"
Sep 11 17:18:40.281901 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.281784    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9202eb18634ea6b37b4f9ec563ddc4eb-config\") pod \"kube-apiserver-proxy-ip-10-0-133-252.ec2.internal\" (UID: \"9202eb18634ea6b37b4f9ec563ddc4eb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-252.ec2.internal"
Sep 11 17:18:40.357341 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:40.357316    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-252.ec2.internal\" not found"
Sep 11 17:18:40.382701 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.382673    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9202eb18634ea6b37b4f9ec563ddc4eb-config\") pod \"kube-apiserver-proxy-ip-10-0-133-252.ec2.internal\" (UID: \"9202eb18634ea6b37b4f9ec563ddc4eb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-252.ec2.internal"
Sep 11 17:18:40.382769 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.382705    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4d9878da489eb3fb19c741850664c940-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-252.ec2.internal\" (UID: \"4d9878da489eb3fb19c741850664c940\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-252.ec2.internal"
Sep 11 17:18:40.382769 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.382721    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d9878da489eb3fb19c741850664c940-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-252.ec2.internal\" (UID: \"4d9878da489eb3fb19c741850664c940\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-252.ec2.internal"
Sep 11 17:18:40.382841 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.382778    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9202eb18634ea6b37b4f9ec563ddc4eb-config\") pod \"kube-apiserver-proxy-ip-10-0-133-252.ec2.internal\" (UID: \"9202eb18634ea6b37b4f9ec563ddc4eb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-252.ec2.internal"
Sep 11 17:18:40.382882 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.382840    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d9878da489eb3fb19c741850664c940-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-252.ec2.internal\" (UID: \"4d9878da489eb3fb19c741850664c940\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-252.ec2.internal"
Sep 11 17:18:40.382916 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.382883    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4d9878da489eb3fb19c741850664c940-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-252.ec2.internal\" (UID: \"4d9878da489eb3fb19c741850664c940\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-252.ec2.internal"
Sep 11 17:18:40.458024 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:40.457983    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-252.ec2.internal\" not found"
Sep 11 17:18:40.548372 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.548339    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-252.ec2.internal"
Sep 11 17:18:40.553061 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.553045    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-252.ec2.internal"
Sep 11 17:18:40.558561 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:40.558542    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-252.ec2.internal\" not found"
Sep 11 17:18:40.658894 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:40.658857    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-252.ec2.internal\" not found"
Sep 11 17:18:40.759286 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:40.759231    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-252.ec2.internal\" not found"
Sep 11 17:18:40.859539 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:40.859515    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-252.ec2.internal\" not found"
Sep 11 17:18:40.910876 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.910854    2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 17:18:40.911252 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.911007    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 17:18:40.911252 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.911029    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 17:18:40.960318 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:40.960294    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-252.ec2.internal\" not found"
Sep 11 17:18:40.976500 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.976454    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 17:13:39 +0000 UTC" deadline="2028-06-08 11:15:48.913594454 +0000 UTC"
Sep 11 17:18:40.976500 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:40.976495    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15257h57m7.937101806s"
Sep 11 17:18:41.043963 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:41.043928    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9202eb18634ea6b37b4f9ec563ddc4eb.slice/crio-901cc480f6887523f2c054012c3c4bdd323b1dec0908da3b041e96f74c722f42 WatchSource:0}: Error finding container 901cc480f6887523f2c054012c3c4bdd323b1dec0908da3b041e96f74c722f42: Status 404 returned error can't find the container with id 901cc480f6887523f2c054012c3c4bdd323b1dec0908da3b041e96f74c722f42
Sep 11 17:18:41.044385 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:41.044345    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d9878da489eb3fb19c741850664c940.slice/crio-a3a10862c6d915cf17f61e1e70cf2a4f6a1d2171c71b78dde5889933b61f815d WatchSource:0}: Error finding container a3a10862c6d915cf17f61e1e70cf2a4f6a1d2171c71b78dde5889933b61f815d: Status 404 returned error can't find the container with id a3a10862c6d915cf17f61e1e70cf2a4f6a1d2171c71b78dde5889933b61f815d
Sep 11 17:18:41.048202 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.048186    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 17:18:41.060886 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:41.060861    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-252.ec2.internal\" not found"
Sep 11 17:18:41.105542 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.105523    2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:18:41.122035 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.121976    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-252.ec2.internal" event={"ID":"4d9878da489eb3fb19c741850664c940","Type":"ContainerStarted","Data":"a3a10862c6d915cf17f61e1e70cf2a4f6a1d2171c71b78dde5889933b61f815d"}
Sep 11 17:18:41.122958 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.122940    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-252.ec2.internal" event={"ID":"9202eb18634ea6b37b4f9ec563ddc4eb","Type":"ContainerStarted","Data":"901cc480f6887523f2c054012c3c4bdd323b1dec0908da3b041e96f74c722f42"}
Sep 11 17:18:41.178047 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.178021    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-252.ec2.internal"
Sep 11 17:18:41.187966 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.187948    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 17:18:41.189579 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.189565    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-252.ec2.internal"
Sep 11 17:18:41.196799 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.196785    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 17:18:41.388596 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.388528    2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:18:41.957226 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.957190    2562 apiserver.go:52] "Watching apiserver"
Sep 11 17:18:41.965880 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.965850    2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 17:18:41.966267 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.966241    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-scdq8","openshift-network-diagnostics/network-check-target-frlv9","openshift-ovn-kubernetes/ovnkube-node-wr5ps","kube-system/global-pull-secret-syncer-4lmxn","kube-system/konnectivity-agent-qjdvg","openshift-dns/node-resolver-km5mq","openshift-multus/multus-8jv92","openshift-multus/multus-additional-cni-plugins-wc7rv","openshift-network-operator/iptables-alerter-rv2lp","kube-system/kube-apiserver-proxy-ip-10-0-133-252.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg9g2","openshift-cluster-node-tuning-operator/tuned-b2jdx","openshift-image-registry/node-ca-2sn6n","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-252.ec2.internal"]
Sep 11 17:18:41.967948 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.967797    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qjdvg"
Sep 11 17:18:41.968791 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.968770    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:41.970047 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.970028    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rv2lp"
Sep 11 17:18:41.970900 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.970882    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-km5mq"
Sep 11 17:18:41.972100 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.972082    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 17:18:41.972248 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.972232    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8jv92"
Sep 11 17:18:41.973536 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.973517    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4lmxn"
Sep 11 17:18:41.973626 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:41.973579    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4lmxn" podUID="85715083-7220-4a0d-9dd0-426a131d8440"
Sep 11 17:18:41.974455 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.974437    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qtt2x\""
Sep 11 17:18:41.974540 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.974444    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 17:18:41.974686 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.974669    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-scdq8"
Sep 11 17:18:41.974796 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:41.974723    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-scdq8" podUID="a382e1b1-e35e-4975-823b-0e9efe9a84f3"
Sep 11 17:18:41.974874 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.974800    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 17:18:41.974874 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.974808    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 17:18:41.975741 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.975445    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 17:18:41.975741 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.975502    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 17:18:41.977451 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.977429    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 17:18:41.978323 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.978306    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 17:18:41.978747 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.978732    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kh2s9\""
Sep 11 17:18:41.979019 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.979006    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 17:18:41.979249 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.979234    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 17:18:41.979549 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.979532    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 17:18:41.979795 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.979778    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bjlc9\""
Sep 11 17:18:41.979982 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.979942    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 17:18:41.980064 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.980048    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-km2wv\""
Sep 11 17:18:41.980348 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.980327    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xm9gh\""
Sep 11 17:18:41.980424 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.980402    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 17:18:41.981352 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.981331    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wc7rv"
Sep 11 17:18:41.981429 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.981416    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 17:18:41.981643 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.981627    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:41.982969 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.982951    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg9g2"
Sep 11 17:18:41.983234 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.983216    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 17:18:41.983766 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.983747    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 17:18:41.984000 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.983965    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 17:18:41.984093 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.984074    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 17:18:41.984232 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.984217    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bp2mg\""
Sep 11 17:18:41.984301 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.984281    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-frlv9"
Sep 11 17:18:41.984354 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:41.984333    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-frlv9" podUID="fab8e5a6-345d-4e9d-984e-3019fb93b9a9"
Sep 11 17:18:41.984487 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.984471    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 17:18:41.984556 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.984477    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 17:18:41.984822 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.984807    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-f42nh\""
Sep 11 17:18:41.984822 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.984817    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 17:18:41.984947 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.984838    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 17:18:41.985026 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.984956    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jcpk2\""
Sep 11 17:18:41.985222 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.985108    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 17:18:41.985402 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.985381    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 17:18:41.985470 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.985402    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 17:18:41.985774 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.985756    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2sn6n"
Sep 11 17:18:41.987479 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.987462    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 17:18:41.987563 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.987524    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 17:18:41.987841 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.987826    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-sjwwm\""
Sep 11 17:18:41.987904 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.987881    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 17:18:41.991460 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.991441    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-host-run-multus-certs\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:41.991543 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.991472    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-etc-kubernetes\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:41.991596 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.991540    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-etc-tuned\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:41.991596 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.991558    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-system-cni-dir\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:41.991596 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.991578    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-multus-socket-dir-parent\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:41.991596 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.991596    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-host-var-lib-cni-bin\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:41.991777 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.991608    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-hostroot\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:41.991777 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.991625    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqprl\" (UniqueName: \"kubernetes.io/projected/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-kube-api-access-kqprl\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:41.991777 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.991645    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/85715083-7220-4a0d-9dd0-426a131d8440-kubelet-config\") pod \"global-pull-secret-syncer-4lmxn\" (UID: \"85715083-7220-4a0d-9dd0-426a131d8440\") " pod="kube-system/global-pull-secret-syncer-4lmxn"
Sep 11 17:18:41.991777 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.991681    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/85715083-7220-4a0d-9dd0-426a131d8440-dbus\") pod \"global-pull-secret-syncer-4lmxn\" (UID: \"85715083-7220-4a0d-9dd0-426a131d8440\") " pod="kube-system/global-pull-secret-syncer-4lmxn"
Sep 11 17:18:41.991777 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.991713    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-host-var-lib-kubelet\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:41.991777 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.991739    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a382e1b1-e35e-4975-823b-0e9efe9a84f3-metrics-certs\") pod \"network-metrics-daemon-scdq8\" (UID: \"a382e1b1-e35e-4975-823b-0e9efe9a84f3\") " pod="openshift-multus/network-metrics-daemon-scdq8"
Sep 11 17:18:41.991777 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.991762    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whv9m\" (UniqueName: \"kubernetes.io/projected/a382e1b1-e35e-4975-823b-0e9efe9a84f3-kube-api-access-whv9m\") pod \"network-metrics-daemon-scdq8\" (UID: \"a382e1b1-e35e-4975-823b-0e9efe9a84f3\") " pod="openshift-multus/network-metrics-daemon-scdq8"
Sep 11 17:18:41.992104 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.991786    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3c72203-332a-4273-8607-6e7f71377e54-tmp-dir\") pod \"node-resolver-km5mq\" (UID: \"c3c72203-332a-4273-8607-6e7f71377e54\") " pod="openshift-dns/node-resolver-km5mq"
Sep 11 17:18:41.992104 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.991807    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8aa0128b-3d1f-4a33-9051-b9eb3edc789b-system-cni-dir\") pod \"multus-additional-cni-plugins-wc7rv\" (UID: \"8aa0128b-3d1f-4a33-9051-b9eb3edc789b\") " pod="openshift-multus/multus-additional-cni-plugins-wc7rv"
Sep 11 17:18:41.992104 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.991828    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8aa0128b-3d1f-4a33-9051-b9eb3edc789b-cnibin\") pod \"multus-additional-cni-plugins-wc7rv\" (UID: \"8aa0128b-3d1f-4a33-9051-b9eb3edc789b\") " pod="openshift-multus/multus-additional-cni-plugins-wc7rv"
Sep 11 17:18:41.992104 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.991850    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-etc-sysctl-d\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:41.992104 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.991878    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-host-run-netns\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:41.992104 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.991901    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85715083-7220-4a0d-9dd0-426a131d8440-original-pull-secret\") pod \"global-pull-secret-syncer-4lmxn\" (UID: \"85715083-7220-4a0d-9dd0-426a131d8440\") " pod="kube-system/global-pull-secret-syncer-4lmxn"
Sep 11 17:18:41.992104 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.991923    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1050a519-3e79-4de7-acb6-a168148cdbc3-iptables-alerter-script\") pod \"iptables-alerter-rv2lp\" (UID: \"1050a519-3e79-4de7-acb6-a168148cdbc3\") " pod="openshift-network-operator/iptables-alerter-rv2lp"
Sep 11 17:18:41.992104 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.991947    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btfmt\" (UniqueName: \"kubernetes.io/projected/1050a519-3e79-4de7-acb6-a168148cdbc3-kube-api-access-btfmt\") pod \"iptables-alerter-rv2lp\" (UID: \"1050a519-3e79-4de7-acb6-a168148cdbc3\") " pod="openshift-network-operator/iptables-alerter-rv2lp"
Sep 11 17:18:41.992104 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.991976    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c3c72203-332a-4273-8607-6e7f71377e54-hosts-file\") pod \"node-resolver-km5mq\" (UID: \"c3c72203-332a-4273-8607-6e7f71377e54\") " pod="openshift-dns/node-resolver-km5mq"
Sep 11 17:18:41.992104 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.992014    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8aa0128b-3d1f-4a33-9051-b9eb3edc789b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wc7rv\" (UID: \"8aa0128b-3d1f-4a33-9051-b9eb3edc789b\") " pod="openshift-multus/multus-additional-cni-plugins-wc7rv"
Sep 11 17:18:41.992104 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.992036    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-sys\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:41.992104 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.992059    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-tmp\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:41.992104 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.992079    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-host-var-lib-cni-multus\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:41.992621 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.992128    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-multus-daemon-config\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:41.992621 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.992160    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8aa0128b-3d1f-4a33-9051-b9eb3edc789b-cni-binary-copy\") pod \"multus-additional-cni-plugins-wc7rv\" (UID: \"8aa0128b-3d1f-4a33-9051-b9eb3edc789b\") " pod="openshift-multus/multus-additional-cni-plugins-wc7rv"
Sep 11 17:18:41.992621 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.992189    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/edbdd0ae-4406-4088-9230-c68aed929e7b-konnectivity-ca\") pod \"konnectivity-agent-qjdvg\" (UID: \"edbdd0ae-4406-4088-9230-c68aed929e7b\") " pod="kube-system/konnectivity-agent-qjdvg"
Sep 11 17:18:41.992621 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.992251    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-etc-kubernetes\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:41.992621 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.992274    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-multus-conf-dir\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:41.992621 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.992293    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8aa0128b-3d1f-4a33-9051-b9eb3edc789b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wc7rv\" (UID: \"8aa0128b-3d1f-4a33-9051-b9eb3edc789b\") " pod="openshift-multus/multus-additional-cni-plugins-wc7rv"
Sep 11 17:18:41.992621 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.992315    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/edbdd0ae-4406-4088-9230-c68aed929e7b-agent-certs\") pod \"konnectivity-agent-qjdvg\" (UID: \"edbdd0ae-4406-4088-9230-c68aed929e7b\") " pod="kube-system/konnectivity-agent-qjdvg"
Sep 11 17:18:41.992621 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.992372    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-run\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:41.992621 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.992394    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-lib-modules\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:41.992621 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.992417    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-var-lib-kubelet\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:41.992621 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.992470    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-multus-cni-dir\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:41.992621 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.992494    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-host-run-k8s-cni-cncf-io\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:41.992621 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.992521    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8aa0128b-3d1f-4a33-9051-b9eb3edc789b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wc7rv\" (UID: \"8aa0128b-3d1f-4a33-9051-b9eb3edc789b\") " pod="openshift-multus/multus-additional-cni-plugins-wc7rv"
Sep 11 17:18:41.992621 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.992547    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-etc-modprobe-d\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:41.992621 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.992566    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-etc-sysctl-conf\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:41.992621 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.992587    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1050a519-3e79-4de7-acb6-a168148cdbc3-host-slash\") pod \"iptables-alerter-rv2lp\" (UID: \"1050a519-3e79-4de7-acb6-a168148cdbc3\") " pod="openshift-network-operator/iptables-alerter-rv2lp"
Sep 11 17:18:41.993336 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.992628    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvs7c\" (UniqueName: \"kubernetes.io/projected/c3c72203-332a-4273-8607-6e7f71377e54-kube-api-access-cvs7c\") pod \"node-resolver-km5mq\" (UID: \"c3c72203-332a-4273-8607-6e7f71377e54\") " pod="openshift-dns/node-resolver-km5mq"
Sep 11 17:18:41.993336 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.992650    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh4lt\" (UniqueName: \"kubernetes.io/projected/8aa0128b-3d1f-4a33-9051-b9eb3edc789b-kube-api-access-mh4lt\") pod \"multus-additional-cni-plugins-wc7rv\" (UID: \"8aa0128b-3d1f-4a33-9051-b9eb3edc789b\") " pod="openshift-multus/multus-additional-cni-plugins-wc7rv"
Sep 11 17:18:41.993336 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.992670    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-etc-sysconfig\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:41.993336 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.992691    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-etc-systemd\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:41.993336 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.992724    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-host\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:41.993336 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.992769    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9st7j\" (UniqueName: \"kubernetes.io/projected/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-kube-api-access-9st7j\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:41.993336 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.992791    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-cnibin\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:41.993336 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:41.992807    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-cni-binary-copy\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:42.000138 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.000121    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 17:18:42.022159 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.022136    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bmsl4"
Sep 11 17:18:42.031235 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.031216    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bmsl4"
Sep 11 17:18:42.033779 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.033763    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:18:42.045015 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.044981    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:18:42.079253 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.079218    2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 17:18:42.093356 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.093325    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-etc-sysctl-conf\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:42.093356 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.093359    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1050a519-3e79-4de7-acb6-a168148cdbc3-host-slash\") pod \"iptables-alerter-rv2lp\" (UID: \"1050a519-3e79-4de7-acb6-a168148cdbc3\") " pod="openshift-network-operator/iptables-alerter-rv2lp"
Sep 11 17:18:42.093541 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.093383    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-run-ovn\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.093541 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.093402    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.093541 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.093425    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/49d0deda-35f9-4165-be19-4c680b5878bd-ovn-node-metrics-cert\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.093541 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.093445    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29n5r\" (UniqueName: \"kubernetes.io/projected/49d0deda-35f9-4165-be19-4c680b5878bd-kube-api-access-29n5r\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.093541 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.093443    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1050a519-3e79-4de7-acb6-a168148cdbc3-host-slash\") pod \"iptables-alerter-rv2lp\" (UID: \"1050a519-3e79-4de7-acb6-a168148cdbc3\") " pod="openshift-network-operator/iptables-alerter-rv2lp"
Sep 11 17:18:42.093541 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.093465    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-etc-systemd\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:42.093541 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.093486    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9st7j\" (UniqueName: \"kubernetes.io/projected/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-kube-api-access-9st7j\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:42.093907 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.093559    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-etc-sysctl-conf\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:42.093907 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.093558    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-etc-systemd\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:42.093907 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.093635    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-cnibin\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:42.093907 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.093660    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-cni-binary-copy\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:42.093907 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.093673    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-host-run-multus-certs\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:42.093907 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.093683    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-cnibin\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:42.093907 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.093687    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-etc-kubernetes\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:42.093907 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.093722    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-etc-kubernetes\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:42.093907 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.093725    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqprl\" (UniqueName: \"kubernetes.io/projected/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-kube-api-access-kqprl\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:42.093907 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.093740    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-host-run-multus-certs\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:42.093907 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.093753    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btfmt\" (UniqueName: \"kubernetes.io/projected/1050a519-3e79-4de7-acb6-a168148cdbc3-kube-api-access-btfmt\") pod \"iptables-alerter-rv2lp\" (UID: \"1050a519-3e79-4de7-acb6-a168148cdbc3\") " pod="openshift-network-operator/iptables-alerter-rv2lp"
Sep 11 17:18:42.093907 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.093776    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-multus-socket-dir-parent\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:42.093907 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.093794    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/85715083-7220-4a0d-9dd0-426a131d8440-kubelet-config\") pod \"global-pull-secret-syncer-4lmxn\" (UID: \"85715083-7220-4a0d-9dd0-426a131d8440\") " pod="kube-system/global-pull-secret-syncer-4lmxn"
Sep 11 17:18:42.093907 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.093839    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/85715083-7220-4a0d-9dd0-426a131d8440-kubelet-config\") pod \"global-pull-secret-syncer-4lmxn\" (UID: \"85715083-7220-4a0d-9dd0-426a131d8440\") " pod="kube-system/global-pull-secret-syncer-4lmxn"
Sep 11 17:18:42.093907 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.093892    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-multus-socket-dir-parent\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:42.094587 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.093927    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/85715083-7220-4a0d-9dd0-426a131d8440-dbus\") pod \"global-pull-secret-syncer-4lmxn\" (UID: \"85715083-7220-4a0d-9dd0-426a131d8440\") " pod="kube-system/global-pull-secret-syncer-4lmxn"
Sep 11 17:18:42.094587 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.093951    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a382e1b1-e35e-4975-823b-0e9efe9a84f3-metrics-certs\") pod \"network-metrics-daemon-scdq8\" (UID: \"a382e1b1-e35e-4975-823b-0e9efe9a84f3\") " pod="openshift-multus/network-metrics-daemon-scdq8"
Sep 11 17:18:42.094587 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.093972    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3c72203-332a-4273-8607-6e7f71377e54-tmp-dir\") pod \"node-resolver-km5mq\" (UID: \"c3c72203-332a-4273-8607-6e7f71377e54\") " pod="openshift-dns/node-resolver-km5mq"
Sep 11 17:18:42.094587 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094014    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8aa0128b-3d1f-4a33-9051-b9eb3edc789b-system-cni-dir\") pod \"multus-additional-cni-plugins-wc7rv\" (UID: \"8aa0128b-3d1f-4a33-9051-b9eb3edc789b\") " pod="openshift-multus/multus-additional-cni-plugins-wc7rv"
Sep 11 17:18:42.094587 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094035    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/85715083-7220-4a0d-9dd0-426a131d8440-dbus\") pod \"global-pull-secret-syncer-4lmxn\" (UID: \"85715083-7220-4a0d-9dd0-426a131d8440\") " pod="kube-system/global-pull-secret-syncer-4lmxn"
Sep 11 17:18:42.094587 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094038    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-host-kubelet\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.094587 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094067    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-run-systemd\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.094587 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:42.094082    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:18:42.094587 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094092    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whv9m\" (UniqueName: \"kubernetes.io/projected/a382e1b1-e35e-4975-823b-0e9efe9a84f3-kube-api-access-whv9m\") pod \"network-metrics-daemon-scdq8\" (UID: \"a382e1b1-e35e-4975-823b-0e9efe9a84f3\") " pod="openshift-multus/network-metrics-daemon-scdq8"
Sep 11 17:18:42.094587 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094115    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8aa0128b-3d1f-4a33-9051-b9eb3edc789b-cnibin\") pod \"multus-additional-cni-plugins-wc7rv\" (UID: \"8aa0128b-3d1f-4a33-9051-b9eb3edc789b\") " pod="openshift-multus/multus-additional-cni-plugins-wc7rv"
Sep 11 17:18:42.094587 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094135    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4cf09e6c-e060-4f10-a0e8-03f505f8e72f-registration-dir\") pod \"aws-ebs-csi-driver-node-fg9g2\" (UID: \"4cf09e6c-e060-4f10-a0e8-03f505f8e72f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg9g2"
Sep 11 17:18:42.094587 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:42.094153    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a382e1b1-e35e-4975-823b-0e9efe9a84f3-metrics-certs podName:a382e1b1-e35e-4975-823b-0e9efe9a84f3 nodeName:}" failed. No retries permitted until 2026-09-11 17:18:42.594134378 +0000 UTC m=+2.988051825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a382e1b1-e35e-4975-823b-0e9efe9a84f3-metrics-certs") pod "network-metrics-daemon-scdq8" (UID: "a382e1b1-e35e-4975-823b-0e9efe9a84f3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:18:42.094587 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094083    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8aa0128b-3d1f-4a33-9051-b9eb3edc789b-system-cni-dir\") pod \"multus-additional-cni-plugins-wc7rv\" (UID: \"8aa0128b-3d1f-4a33-9051-b9eb3edc789b\") " pod="openshift-multus/multus-additional-cni-plugins-wc7rv"
Sep 11 17:18:42.094587 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094197    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4cf09e6c-e060-4f10-a0e8-03f505f8e72f-sys-fs\") pod \"aws-ebs-csi-driver-node-fg9g2\" (UID: \"4cf09e6c-e060-4f10-a0e8-03f505f8e72f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg9g2"
Sep 11 17:18:42.094587 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094221    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gflqm\" (UniqueName: \"kubernetes.io/projected/924a846a-540b-48c3-a5d7-8df3fbaa41e8-kube-api-access-gflqm\") pod \"node-ca-2sn6n\" (UID: \"924a846a-540b-48c3-a5d7-8df3fbaa41e8\") " pod="openshift-image-registry/node-ca-2sn6n"
Sep 11 17:18:42.094587 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094234    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-cni-binary-copy\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:42.094587 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094245    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-host-run-netns\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:42.095252 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094268    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c3c72203-332a-4273-8607-6e7f71377e54-hosts-file\") pod \"node-resolver-km5mq\" (UID: \"c3c72203-332a-4273-8607-6e7f71377e54\") " pod="openshift-dns/node-resolver-km5mq"
Sep 11 17:18:42.095252 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094275    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3c72203-332a-4273-8607-6e7f71377e54-tmp-dir\") pod \"node-resolver-km5mq\" (UID: \"c3c72203-332a-4273-8607-6e7f71377e54\") " pod="openshift-dns/node-resolver-km5mq"
Sep 11 17:18:42.095252 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094302    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-host-run-netns\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:42.095252 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094302    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8aa0128b-3d1f-4a33-9051-b9eb3edc789b-cnibin\") pod \"multus-additional-cni-plugins-wc7rv\" (UID: \"8aa0128b-3d1f-4a33-9051-b9eb3edc789b\") " pod="openshift-multus/multus-additional-cni-plugins-wc7rv"
Sep 11 17:18:42.095252 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094320    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8aa0128b-3d1f-4a33-9051-b9eb3edc789b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wc7rv\" (UID: \"8aa0128b-3d1f-4a33-9051-b9eb3edc789b\") " pod="openshift-multus/multus-additional-cni-plugins-wc7rv"
Sep 11 17:18:42.095252 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094329    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c3c72203-332a-4273-8607-6e7f71377e54-hosts-file\") pod \"node-resolver-km5mq\" (UID: \"c3c72203-332a-4273-8607-6e7f71377e54\") " pod="openshift-dns/node-resolver-km5mq"
Sep 11 17:18:42.095252 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094342    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-var-lib-openvswitch\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.095252 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094357    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-etc-openvswitch\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.095252 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094379    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-run-openvswitch\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.095252 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094400    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-host-cni-netd\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.095252 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094422    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/49d0deda-35f9-4165-be19-4c680b5878bd-ovnkube-script-lib\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.095252 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094443    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-tmp\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:42.095252 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094464    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-multus-daemon-config\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:42.095252 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094485    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8aa0128b-3d1f-4a33-9051-b9eb3edc789b-cni-binary-copy\") pod \"multus-additional-cni-plugins-wc7rv\" (UID: \"8aa0128b-3d1f-4a33-9051-b9eb3edc789b\") " pod="openshift-multus/multus-additional-cni-plugins-wc7rv"
Sep 11 17:18:42.095252 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094502    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-log-socket\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.095252 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094629    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjvvx\" (UniqueName: \"kubernetes.io/projected/fab8e5a6-345d-4e9d-984e-3019fb93b9a9-kube-api-access-mjvvx\") pod \"network-check-target-frlv9\" (UID: \"fab8e5a6-345d-4e9d-984e-3019fb93b9a9\") " pod="openshift-network-diagnostics/network-check-target-frlv9"
Sep 11 17:18:42.095252 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094658    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/924a846a-540b-48c3-a5d7-8df3fbaa41e8-host\") pod \"node-ca-2sn6n\" (UID: \"924a846a-540b-48c3-a5d7-8df3fbaa41e8\") " pod="openshift-image-registry/node-ca-2sn6n"
Sep 11 17:18:42.095887 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094694    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/924a846a-540b-48c3-a5d7-8df3fbaa41e8-serviceca\") pod \"node-ca-2sn6n\" (UID: \"924a846a-540b-48c3-a5d7-8df3fbaa41e8\") " pod="openshift-image-registry/node-ca-2sn6n"
Sep 11 17:18:42.095887 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094722    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/edbdd0ae-4406-4088-9230-c68aed929e7b-konnectivity-ca\") pod \"konnectivity-agent-qjdvg\" (UID: \"edbdd0ae-4406-4088-9230-c68aed929e7b\") " pod="kube-system/konnectivity-agent-qjdvg"
Sep 11 17:18:42.095887 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094744    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-etc-kubernetes\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:42.095887 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094766    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4cf09e6c-e060-4f10-a0e8-03f505f8e72f-device-dir\") pod \"aws-ebs-csi-driver-node-fg9g2\" (UID: \"4cf09e6c-e060-4f10-a0e8-03f505f8e72f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg9g2"
Sep 11 17:18:42.095887 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094810    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8aa0128b-3d1f-4a33-9051-b9eb3edc789b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wc7rv\" (UID: \"8aa0128b-3d1f-4a33-9051-b9eb3edc789b\") " pod="openshift-multus/multus-additional-cni-plugins-wc7rv"
Sep 11 17:18:42.095887 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094821    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-run\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:42.095887 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094844    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-multus-cni-dir\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:42.095887 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094866    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-host-run-k8s-cni-cncf-io\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:42.095887 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094885    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-systemd-units\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.095887 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094915    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-etc-modprobe-d\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:42.095887 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094923    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-multus-daemon-config\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:42.095887 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094937    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvs7c\" (UniqueName: \"kubernetes.io/projected/c3c72203-332a-4273-8607-6e7f71377e54-kube-api-access-cvs7c\") pod \"node-resolver-km5mq\" (UID: \"c3c72203-332a-4273-8607-6e7f71377e54\") " pod="openshift-dns/node-resolver-km5mq"
Sep 11 17:18:42.095887 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094931    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-etc-kubernetes\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:42.095887 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094947    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8aa0128b-3d1f-4a33-9051-b9eb3edc789b-cni-binary-copy\") pod \"multus-additional-cni-plugins-wc7rv\" (UID: \"8aa0128b-3d1f-4a33-9051-b9eb3edc789b\") " pod="openshift-multus/multus-additional-cni-plugins-wc7rv"
Sep 11 17:18:42.095887 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094952    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh4lt\" (UniqueName: \"kubernetes.io/projected/8aa0128b-3d1f-4a33-9051-b9eb3edc789b-kube-api-access-mh4lt\") pod \"multus-additional-cni-plugins-wc7rv\" (UID: \"8aa0128b-3d1f-4a33-9051-b9eb3edc789b\") " pod="openshift-multus/multus-additional-cni-plugins-wc7rv"
Sep 11 17:18:42.095887 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.094978    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-host-run-k8s-cni-cncf-io\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:42.095887 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095008    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4cf09e6c-e060-4f10-a0e8-03f505f8e72f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fg9g2\" (UID: \"4cf09e6c-e060-4f10-a0e8-03f505f8e72f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg9g2"
Sep 11 17:18:42.096626 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095036    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-run\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:42.096626 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095036    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-etc-modprobe-d\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:42.096626 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095036    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-etc-sysconfig\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:42.096626 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095083    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-host\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:42.096626 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095084    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-etc-sysconfig\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:42.096626 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095091    2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 17:18:42.096626 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095120    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85715083-7220-4a0d-9dd0-426a131d8440-original-pull-secret\") pod \"global-pull-secret-syncer-4lmxn\" (UID: \"85715083-7220-4a0d-9dd0-426a131d8440\") " pod="kube-system/global-pull-secret-syncer-4lmxn"
Sep 11 17:18:42.096626 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095127    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-multus-cni-dir\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:42.096626 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095145    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1050a519-3e79-4de7-acb6-a168148cdbc3-iptables-alerter-script\") pod \"iptables-alerter-rv2lp\" (UID: \"1050a519-3e79-4de7-acb6-a168148cdbc3\") " pod="openshift-network-operator/iptables-alerter-rv2lp"
Sep 11 17:18:42.096626 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095178    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-host\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:42.096626 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:42.095181    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:18:42.096626 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095214    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-host-run-ovn-kubernetes\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.096626 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095245    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/edbdd0ae-4406-4088-9230-c68aed929e7b-konnectivity-ca\") pod \"konnectivity-agent-qjdvg\" (UID: \"edbdd0ae-4406-4088-9230-c68aed929e7b\") " pod="kube-system/konnectivity-agent-qjdvg"
Sep 11 17:18:42.096626 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:42.095277    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85715083-7220-4a0d-9dd0-426a131d8440-original-pull-secret podName:85715083-7220-4a0d-9dd0-426a131d8440 nodeName:}" failed. No retries permitted until 2026-09-11 17:18:42.595261984 +0000 UTC m=+2.989179422 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/85715083-7220-4a0d-9dd0-426a131d8440-original-pull-secret") pod "global-pull-secret-syncer-4lmxn" (UID: "85715083-7220-4a0d-9dd0-426a131d8440") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:18:42.096626 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095300    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/49d0deda-35f9-4165-be19-4c680b5878bd-env-overrides\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.096626 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095369    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-etc-tuned\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:42.096626 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095395    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-system-cni-dir\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:42.096626 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095417    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-host-var-lib-cni-bin\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:42.097321 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095444    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-hostroot\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:42.097321 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095497    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-system-cni-dir\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:42.097321 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095496    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-host-var-lib-cni-bin\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:42.097321 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095519    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-host-var-lib-kubelet\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:42.097321 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095538    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-hostroot\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:42.097321 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095546    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-node-log\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.097321 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095569    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/49d0deda-35f9-4165-be19-4c680b5878bd-ovnkube-config\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.097321 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095581    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-host-var-lib-kubelet\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:42.097321 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095590    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4cf09e6c-e060-4f10-a0e8-03f505f8e72f-socket-dir\") pod \"aws-ebs-csi-driver-node-fg9g2\" (UID: \"4cf09e6c-e060-4f10-a0e8-03f505f8e72f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg9g2"
Sep 11 17:18:42.097321 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095611    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-host-slash\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.097321 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095637    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-etc-sysctl-d\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:42.097321 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095662    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-host-run-netns\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.097321 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095682    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-sys\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:42.097321 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095699    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1050a519-3e79-4de7-acb6-a168148cdbc3-iptables-alerter-script\") pod \"iptables-alerter-rv2lp\" (UID: \"1050a519-3e79-4de7-acb6-a168148cdbc3\") " pod="openshift-network-operator/iptables-alerter-rv2lp"
Sep 11 17:18:42.097321 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095701    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-host-var-lib-cni-multus\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:42.097321 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095726    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-host-var-lib-cni-multus\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:42.097321 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095741    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4cf09e6c-e060-4f10-a0e8-03f505f8e72f-etc-selinux\") pod \"aws-ebs-csi-driver-node-fg9g2\" (UID: \"4cf09e6c-e060-4f10-a0e8-03f505f8e72f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg9g2"
Sep 11 17:18:42.097825 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095755    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-etc-sysctl-d\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:42.097825 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095762    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvm6g\" (UniqueName: \"kubernetes.io/projected/4cf09e6c-e060-4f10-a0e8-03f505f8e72f-kube-api-access-gvm6g\") pod \"aws-ebs-csi-driver-node-fg9g2\" (UID: \"4cf09e6c-e060-4f10-a0e8-03f505f8e72f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg9g2"
Sep 11 17:18:42.097825 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095768    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-sys\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:42.097825 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095798    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-multus-conf-dir\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:42.097825 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095813    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8aa0128b-3d1f-4a33-9051-b9eb3edc789b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wc7rv\" (UID: \"8aa0128b-3d1f-4a33-9051-b9eb3edc789b\") " pod="openshift-multus/multus-additional-cni-plugins-wc7rv"
Sep 11 17:18:42.097825 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095828    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/edbdd0ae-4406-4088-9230-c68aed929e7b-agent-certs\") pod \"konnectivity-agent-qjdvg\" (UID: \"edbdd0ae-4406-4088-9230-c68aed929e7b\") " pod="kube-system/konnectivity-agent-qjdvg"
Sep 11 17:18:42.097825 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095841    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-lib-modules\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:42.097825 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095856    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-var-lib-kubelet\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:42.097825 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095876    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8aa0128b-3d1f-4a33-9051-b9eb3edc789b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wc7rv\" (UID: \"8aa0128b-3d1f-4a33-9051-b9eb3edc789b\") " pod="openshift-multus/multus-additional-cni-plugins-wc7rv"
Sep 11 17:18:42.097825 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095913    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-host-cni-bin\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.097825 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095965    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8aa0128b-3d1f-4a33-9051-b9eb3edc789b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wc7rv\" (UID: \"8aa0128b-3d1f-4a33-9051-b9eb3edc789b\") " pod="openshift-multus/multus-additional-cni-plugins-wc7rv"
Sep 11 17:18:42.097825 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.095839    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-multus-conf-dir\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:42.097825 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.096085    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-lib-modules\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:42.097825 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.096135    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-var-lib-kubelet\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:42.097825 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.096542    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8aa0128b-3d1f-4a33-9051-b9eb3edc789b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wc7rv\" (UID: \"8aa0128b-3d1f-4a33-9051-b9eb3edc789b\") " pod="openshift-multus/multus-additional-cni-plugins-wc7rv"
Sep 11 17:18:42.098745 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.098725    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-etc-tuned\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:42.098834 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.098745    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-tmp\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:42.098889 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.098845    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/edbdd0ae-4406-4088-9230-c68aed929e7b-agent-certs\") pod \"konnectivity-agent-qjdvg\" (UID: \"edbdd0ae-4406-4088-9230-c68aed929e7b\") " pod="kube-system/konnectivity-agent-qjdvg"
Sep 11 17:18:42.103080 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.103064    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btfmt\" (UniqueName: \"kubernetes.io/projected/1050a519-3e79-4de7-acb6-a168148cdbc3-kube-api-access-btfmt\") pod \"iptables-alerter-rv2lp\" (UID: \"1050a519-3e79-4de7-acb6-a168148cdbc3\") " pod="openshift-network-operator/iptables-alerter-rv2lp"
Sep 11 17:18:42.107662 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.107641    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqprl\" (UniqueName: \"kubernetes.io/projected/22a4137e-aa6e-4cbd-bd97-8e252cfb41bd-kube-api-access-kqprl\") pod \"multus-8jv92\" (UID: \"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd\") " pod="openshift-multus/multus-8jv92"
Sep 11 17:18:42.108279 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.108219    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9st7j\" (UniqueName: \"kubernetes.io/projected/bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6-kube-api-access-9st7j\") pod \"tuned-b2jdx\" (UID: \"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6\") " pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:42.108654 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.108625    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvs7c\" (UniqueName: \"kubernetes.io/projected/c3c72203-332a-4273-8607-6e7f71377e54-kube-api-access-cvs7c\") pod \"node-resolver-km5mq\" (UID: \"c3c72203-332a-4273-8607-6e7f71377e54\") " pod="openshift-dns/node-resolver-km5mq"
Sep 11 17:18:42.108781 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.108764    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh4lt\" (UniqueName: \"kubernetes.io/projected/8aa0128b-3d1f-4a33-9051-b9eb3edc789b-kube-api-access-mh4lt\") pod \"multus-additional-cni-plugins-wc7rv\" (UID: \"8aa0128b-3d1f-4a33-9051-b9eb3edc789b\") " pod="openshift-multus/multus-additional-cni-plugins-wc7rv"
Sep 11 17:18:42.109352 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.109338    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whv9m\" (UniqueName: \"kubernetes.io/projected/a382e1b1-e35e-4975-823b-0e9efe9a84f3-kube-api-access-whv9m\") pod \"network-metrics-daemon-scdq8\" (UID: \"a382e1b1-e35e-4975-823b-0e9efe9a84f3\") " pod="openshift-multus/network-metrics-daemon-scdq8"
Sep 11 17:18:42.196904 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.196877    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-systemd-units\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.197067 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.196915    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4cf09e6c-e060-4f10-a0e8-03f505f8e72f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fg9g2\" (UID: \"4cf09e6c-e060-4f10-a0e8-03f505f8e72f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg9g2"
Sep 11 17:18:42.197067 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197012    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4cf09e6c-e060-4f10-a0e8-03f505f8e72f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fg9g2\" (UID: \"4cf09e6c-e060-4f10-a0e8-03f505f8e72f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg9g2"
Sep 11 17:18:42.197067 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.196988    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-systemd-units\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.197067 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197056    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-host-run-ovn-kubernetes\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.197274 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197089    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/49d0deda-35f9-4165-be19-4c680b5878bd-env-overrides\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.197274 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197130    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-node-log\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.197274 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197136    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-host-run-ovn-kubernetes\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.197274 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197152    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/49d0deda-35f9-4165-be19-4c680b5878bd-ovnkube-config\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.197274 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197178    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-node-log\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.197274 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197176    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4cf09e6c-e060-4f10-a0e8-03f505f8e72f-socket-dir\") pod \"aws-ebs-csi-driver-node-fg9g2\" (UID: \"4cf09e6c-e060-4f10-a0e8-03f505f8e72f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg9g2"
Sep 11 17:18:42.197274 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197205    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-host-slash\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.197274 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197224    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-host-run-netns\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.197274 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197249    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4cf09e6c-e060-4f10-a0e8-03f505f8e72f-etc-selinux\") pod \"aws-ebs-csi-driver-node-fg9g2\" (UID: \"4cf09e6c-e060-4f10-a0e8-03f505f8e72f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg9g2"
Sep 11 17:18:42.197668 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197282    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvm6g\" (UniqueName: \"kubernetes.io/projected/4cf09e6c-e060-4f10-a0e8-03f505f8e72f-kube-api-access-gvm6g\") pod \"aws-ebs-csi-driver-node-fg9g2\" (UID: \"4cf09e6c-e060-4f10-a0e8-03f505f8e72f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg9g2"
Sep 11 17:18:42.197668 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197284    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-host-slash\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.197668 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197311    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-host-cni-bin\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.197668 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197336    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-run-ovn\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.197668 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197343    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4cf09e6c-e060-4f10-a0e8-03f505f8e72f-socket-dir\") pod \"aws-ebs-csi-driver-node-fg9g2\" (UID: \"4cf09e6c-e060-4f10-a0e8-03f505f8e72f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg9g2"
Sep 11 17:18:42.197668 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197357    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.197668 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197382    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/49d0deda-35f9-4165-be19-4c680b5878bd-ovn-node-metrics-cert\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.197668 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197385    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-host-cni-bin\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.197668 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197390    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4cf09e6c-e060-4f10-a0e8-03f505f8e72f-etc-selinux\") pod \"aws-ebs-csi-driver-node-fg9g2\" (UID: \"4cf09e6c-e060-4f10-a0e8-03f505f8e72f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg9g2"
Sep 11 17:18:42.197668 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197404    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29n5r\" (UniqueName: \"kubernetes.io/projected/49d0deda-35f9-4165-be19-4c680b5878bd-kube-api-access-29n5r\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.197668 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197426    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.197668 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197450    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-host-kubelet\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.197668 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197461    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-run-ovn\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.197668 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197471    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-run-systemd\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.197668 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197494    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4cf09e6c-e060-4f10-a0e8-03f505f8e72f-registration-dir\") pod \"aws-ebs-csi-driver-node-fg9g2\" (UID: \"4cf09e6c-e060-4f10-a0e8-03f505f8e72f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg9g2"
Sep 11 17:18:42.197668 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197516    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4cf09e6c-e060-4f10-a0e8-03f505f8e72f-sys-fs\") pod \"aws-ebs-csi-driver-node-fg9g2\" (UID: \"4cf09e6c-e060-4f10-a0e8-03f505f8e72f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg9g2"
Sep 11 17:18:42.197668 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197516    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/49d0deda-35f9-4165-be19-4c680b5878bd-env-overrides\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.198317 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197540    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gflqm\" (UniqueName: \"kubernetes.io/projected/924a846a-540b-48c3-a5d7-8df3fbaa41e8-kube-api-access-gflqm\") pod \"node-ca-2sn6n\" (UID: \"924a846a-540b-48c3-a5d7-8df3fbaa41e8\") " pod="openshift-image-registry/node-ca-2sn6n"
Sep 11 17:18:42.198317 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197568    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-var-lib-openvswitch\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.198317 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197618    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-var-lib-openvswitch\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.198317 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197628    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-host-kubelet\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.198317 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197634    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4cf09e6c-e060-4f10-a0e8-03f505f8e72f-sys-fs\") pod \"aws-ebs-csi-driver-node-fg9g2\" (UID: \"4cf09e6c-e060-4f10-a0e8-03f505f8e72f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg9g2"
Sep 11 17:18:42.198317 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197653    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/49d0deda-35f9-4165-be19-4c680b5878bd-ovnkube-config\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.198317 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197662    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-run-systemd\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.198317 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197658    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-host-run-netns\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.198317 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197706    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-etc-openvswitch\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.198317 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197732    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-run-openvswitch\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.198317 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197742    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4cf09e6c-e060-4f10-a0e8-03f505f8e72f-registration-dir\") pod \"aws-ebs-csi-driver-node-fg9g2\" (UID: \"4cf09e6c-e060-4f10-a0e8-03f505f8e72f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg9g2"
Sep 11 17:18:42.198317 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197754    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-etc-openvswitch\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.198317 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197757    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-host-cni-netd\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.198317 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197784    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-run-openvswitch\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.198317 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197789    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-host-cni-netd\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.198317 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197812    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/49d0deda-35f9-4165-be19-4c680b5878bd-ovnkube-script-lib\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.198317 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197840    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-log-socket\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.198813 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197870    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjvvx\" (UniqueName: \"kubernetes.io/projected/fab8e5a6-345d-4e9d-984e-3019fb93b9a9-kube-api-access-mjvvx\") pod \"network-check-target-frlv9\" (UID: \"fab8e5a6-345d-4e9d-984e-3019fb93b9a9\") " pod="openshift-network-diagnostics/network-check-target-frlv9"
Sep 11 17:18:42.198813 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197892    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/924a846a-540b-48c3-a5d7-8df3fbaa41e8-host\") pod \"node-ca-2sn6n\" (UID: \"924a846a-540b-48c3-a5d7-8df3fbaa41e8\") " pod="openshift-image-registry/node-ca-2sn6n"
Sep 11 17:18:42.198813 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197910    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/49d0deda-35f9-4165-be19-4c680b5878bd-log-socket\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.198813 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197929    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/924a846a-540b-48c3-a5d7-8df3fbaa41e8-host\") pod \"node-ca-2sn6n\" (UID: \"924a846a-540b-48c3-a5d7-8df3fbaa41e8\") " pod="openshift-image-registry/node-ca-2sn6n"
Sep 11 17:18:42.198813 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.197952    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/924a846a-540b-48c3-a5d7-8df3fbaa41e8-serviceca\") pod \"node-ca-2sn6n\" (UID: \"924a846a-540b-48c3-a5d7-8df3fbaa41e8\") " pod="openshift-image-registry/node-ca-2sn6n"
Sep 11 17:18:42.198813 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.198005    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4cf09e6c-e060-4f10-a0e8-03f505f8e72f-device-dir\") pod \"aws-ebs-csi-driver-node-fg9g2\" (UID: \"4cf09e6c-e060-4f10-a0e8-03f505f8e72f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg9g2"
Sep 11 17:18:42.198813 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.198050    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4cf09e6c-e060-4f10-a0e8-03f505f8e72f-device-dir\") pod \"aws-ebs-csi-driver-node-fg9g2\" (UID: \"4cf09e6c-e060-4f10-a0e8-03f505f8e72f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg9g2"
Sep 11 17:18:42.199078 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.198818    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/49d0deda-35f9-4165-be19-4c680b5878bd-ovnkube-script-lib\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.199078 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.198885    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/924a846a-540b-48c3-a5d7-8df3fbaa41e8-serviceca\") pod \"node-ca-2sn6n\" (UID: \"924a846a-540b-48c3-a5d7-8df3fbaa41e8\") " pod="openshift-image-registry/node-ca-2sn6n"
Sep 11 17:18:42.199688 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.199665    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/49d0deda-35f9-4165-be19-4c680b5878bd-ovn-node-metrics-cert\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.205722 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:42.205702    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:18:42.205811 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:42.205728    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:18:42.205811 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:42.205741    2562 projected.go:196] Error preparing data for projected volume kube-api-access-mjvvx for pod openshift-network-diagnostics/network-check-target-frlv9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:18:42.205811 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:42.205796    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fab8e5a6-345d-4e9d-984e-3019fb93b9a9-kube-api-access-mjvvx podName:fab8e5a6-345d-4e9d-984e-3019fb93b9a9 nodeName:}" failed. No retries permitted until 2026-09-11 17:18:42.705783766 +0000 UTC m=+3.099701210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mjvvx" (UniqueName: "kubernetes.io/projected/fab8e5a6-345d-4e9d-984e-3019fb93b9a9-kube-api-access-mjvvx") pod "network-check-target-frlv9" (UID: "fab8e5a6-345d-4e9d-984e-3019fb93b9a9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:18:42.206485 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.206467    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvm6g\" (UniqueName: \"kubernetes.io/projected/4cf09e6c-e060-4f10-a0e8-03f505f8e72f-kube-api-access-gvm6g\") pod \"aws-ebs-csi-driver-node-fg9g2\" (UID: \"4cf09e6c-e060-4f10-a0e8-03f505f8e72f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg9g2"
Sep 11 17:18:42.208564 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.208523    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gflqm\" (UniqueName: \"kubernetes.io/projected/924a846a-540b-48c3-a5d7-8df3fbaa41e8-kube-api-access-gflqm\") pod \"node-ca-2sn6n\" (UID: \"924a846a-540b-48c3-a5d7-8df3fbaa41e8\") " pod="openshift-image-registry/node-ca-2sn6n"
Sep 11 17:18:42.208646 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.208588    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29n5r\" (UniqueName: \"kubernetes.io/projected/49d0deda-35f9-4165-be19-4c680b5878bd-kube-api-access-29n5r\") pod \"ovnkube-node-wr5ps\" (UID: \"49d0deda-35f9-4165-be19-4c680b5878bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.281619 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.281590    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qjdvg"
Sep 11 17:18:42.289282 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.289254    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b2jdx"
Sep 11 17:18:42.296901 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.296883    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rv2lp"
Sep 11 17:18:42.300906 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.300884    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-km5mq"
Sep 11 17:18:42.307056 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.307038    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8jv92"
Sep 11 17:18:42.313580 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.313556    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wc7rv"
Sep 11 17:18:42.320067 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.320048    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:18:42.327497 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.327478    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg9g2"
Sep 11 17:18:42.333917 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.333891    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2sn6n"
Sep 11 17:18:42.600481 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.600394    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a382e1b1-e35e-4975-823b-0e9efe9a84f3-metrics-certs\") pod \"network-metrics-daemon-scdq8\" (UID: \"a382e1b1-e35e-4975-823b-0e9efe9a84f3\") " pod="openshift-multus/network-metrics-daemon-scdq8"
Sep 11 17:18:42.600644 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:42.600551    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:18:42.600644 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.600566    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85715083-7220-4a0d-9dd0-426a131d8440-original-pull-secret\") pod \"global-pull-secret-syncer-4lmxn\" (UID: \"85715083-7220-4a0d-9dd0-426a131d8440\") " pod="kube-system/global-pull-secret-syncer-4lmxn"
Sep 11 17:18:42.600644 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:42.600621    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a382e1b1-e35e-4975-823b-0e9efe9a84f3-metrics-certs podName:a382e1b1-e35e-4975-823b-0e9efe9a84f3 nodeName:}" failed. No retries permitted until 2026-09-11 17:18:43.600600921 +0000 UTC m=+3.994518377 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a382e1b1-e35e-4975-823b-0e9efe9a84f3-metrics-certs") pod "network-metrics-daemon-scdq8" (UID: "a382e1b1-e35e-4975-823b-0e9efe9a84f3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:18:42.600824 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:42.600664    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:18:42.600824 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:42.600743    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85715083-7220-4a0d-9dd0-426a131d8440-original-pull-secret podName:85715083-7220-4a0d-9dd0-426a131d8440 nodeName:}" failed. No retries permitted until 2026-09-11 17:18:43.600729176 +0000 UTC m=+3.994646614 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/85715083-7220-4a0d-9dd0-426a131d8440-original-pull-secret") pod "global-pull-secret-syncer-4lmxn" (UID: "85715083-7220-4a0d-9dd0-426a131d8440") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:18:42.801537 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:42.801507    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjvvx\" (UniqueName: \"kubernetes.io/projected/fab8e5a6-345d-4e9d-984e-3019fb93b9a9-kube-api-access-mjvvx\") pod \"network-check-target-frlv9\" (UID: \"fab8e5a6-345d-4e9d-984e-3019fb93b9a9\") " pod="openshift-network-diagnostics/network-check-target-frlv9"
Sep 11 17:18:42.801692 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:42.801644    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:18:42.801692 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:42.801660    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:18:42.801692 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:42.801669    2562 projected.go:196] Error preparing data for projected volume kube-api-access-mjvvx for pod openshift-network-diagnostics/network-check-target-frlv9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:18:42.801808 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:42.801718    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fab8e5a6-345d-4e9d-984e-3019fb93b9a9-kube-api-access-mjvvx podName:fab8e5a6-345d-4e9d-984e-3019fb93b9a9 nodeName:}" failed. No retries permitted until 2026-09-11 17:18:43.801705509 +0000 UTC m=+4.195622942 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mjvvx" (UniqueName: "kubernetes.io/projected/fab8e5a6-345d-4e9d-984e-3019fb93b9a9-kube-api-access-mjvvx") pod "network-check-target-frlv9" (UID: "fab8e5a6-345d-4e9d-984e-3019fb93b9a9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:18:42.822538 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:42.822325    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d0deda_35f9_4165_be19_4c680b5878bd.slice/crio-a97fbdd4127c9ce5c9220bc37ba9044ec1ed6541fafff93353222ed235e4ebea WatchSource:0}: Error finding container a97fbdd4127c9ce5c9220bc37ba9044ec1ed6541fafff93353222ed235e4ebea: Status 404 returned error can't find the container with id a97fbdd4127c9ce5c9220bc37ba9044ec1ed6541fafff93353222ed235e4ebea
Sep 11 17:18:42.824330 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:42.824298    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd7d3816_e8d5_4f64_8c24_4ea88a02f4d6.slice/crio-676e4181b41990fd2e8924c299f910fc7b0563015891ed0d229b9f78aaa522de WatchSource:0}: Error finding container 676e4181b41990fd2e8924c299f910fc7b0563015891ed0d229b9f78aaa522de: Status 404 returned error can't find the container with id 676e4181b41990fd2e8924c299f910fc7b0563015891ed0d229b9f78aaa522de
Sep 11 17:18:42.826495 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:42.826373    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a4137e_aa6e_4cbd_bd97_8e252cfb41bd.slice/crio-a39900ebcc258ffc3febac94a12152f79e448b1a2ae43b2e4a3aadf5671ee9cb WatchSource:0}: Error finding container a39900ebcc258ffc3febac94a12152f79e448b1a2ae43b2e4a3aadf5671ee9cb: Status 404 returned error can't find the container with id a39900ebcc258ffc3febac94a12152f79e448b1a2ae43b2e4a3aadf5671ee9cb
Sep 11 17:18:42.828728 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:42.828708    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedbdd0ae_4406_4088_9230_c68aed929e7b.slice/crio-1d31f59e49dc55421584b1c7dcd46327281669506464355bee94df48855332e2 WatchSource:0}: Error finding container 1d31f59e49dc55421584b1c7dcd46327281669506464355bee94df48855332e2: Status 404 returned error can't find the container with id 1d31f59e49dc55421584b1c7dcd46327281669506464355bee94df48855332e2
Sep 11 17:18:42.829231 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:42.829212    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3c72203_332a_4273_8607_6e7f71377e54.slice/crio-5c3d7928b1ffbdc6e009c09679f2642f58ce81f599c0563ccff00c32521a68d9 WatchSource:0}: Error finding container 5c3d7928b1ffbdc6e009c09679f2642f58ce81f599c0563ccff00c32521a68d9: Status 404 returned error can't find the container with id 5c3d7928b1ffbdc6e009c09679f2642f58ce81f599c0563ccff00c32521a68d9
Sep 11 17:18:42.830739 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:42.830719    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1050a519_3e79_4de7_acb6_a168148cdbc3.slice/crio-49bb3146857c83ac2e2487d8fdf90e4595fe7b57052ee8d0f883ff75ca04cbbb WatchSource:0}: Error finding container 49bb3146857c83ac2e2487d8fdf90e4595fe7b57052ee8d0f883ff75ca04cbbb: Status 404 returned error can't find the container with id 49bb3146857c83ac2e2487d8fdf90e4595fe7b57052ee8d0f883ff75ca04cbbb
Sep 11 17:18:42.832476 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:42.832157    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cf09e6c_e060_4f10_a0e8_03f505f8e72f.slice/crio-4afdaff3149c1584ec0af2800b79f31220e028b75a32907b35c6056f8b036fb9 WatchSource:0}: Error finding container 4afdaff3149c1584ec0af2800b79f31220e028b75a32907b35c6056f8b036fb9: Status 404 returned error can't find the container with id 4afdaff3149c1584ec0af2800b79f31220e028b75a32907b35c6056f8b036fb9
Sep 11 17:18:42.833231 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:42.833193    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa0128b_3d1f_4a33_9051_b9eb3edc789b.slice/crio-0a31dff976bcdf0a3ce63e3509e6f77f2b651456cf80a85fbb31854ccd40ebc4 WatchSource:0}: Error finding container 0a31dff976bcdf0a3ce63e3509e6f77f2b651456cf80a85fbb31854ccd40ebc4: Status 404 returned error can't find the container with id 0a31dff976bcdf0a3ce63e3509e6f77f2b651456cf80a85fbb31854ccd40ebc4
Sep 11 17:18:42.834440 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:18:42.834413    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod924a846a_540b_48c3_a5d7_8df3fbaa41e8.slice/crio-8d1dfa6e422f66b1a6378e99585f8118645b43f91ea2c38a3dd85fa9a5a06606 WatchSource:0}: Error finding container 8d1dfa6e422f66b1a6378e99585f8118645b43f91ea2c38a3dd85fa9a5a06606: Status 404 returned error can't find the container with id 8d1dfa6e422f66b1a6378e99585f8118645b43f91ea2c38a3dd85fa9a5a06606
Sep 11 17:18:43.034769 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:43.034721    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 17:13:41 +0000 UTC" deadline="2028-04-01 07:36:38.112291823 +0000 UTC"
Sep 11 17:18:43.034769 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:43.034765    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13622h17m55.077529157s"
Sep 11 17:18:43.120122 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:43.120057    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-scdq8"
Sep 11 17:18:43.120239 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:43.120195    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-scdq8" podUID="a382e1b1-e35e-4975-823b-0e9efe9a84f3"
Sep 11 17:18:43.127532 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:43.127500    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wc7rv" event={"ID":"8aa0128b-3d1f-4a33-9051-b9eb3edc789b","Type":"ContainerStarted","Data":"0a31dff976bcdf0a3ce63e3509e6f77f2b651456cf80a85fbb31854ccd40ebc4"}
Sep 11 17:18:43.128801 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:43.128780    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg9g2" event={"ID":"4cf09e6c-e060-4f10-a0e8-03f505f8e72f","Type":"ContainerStarted","Data":"4afdaff3149c1584ec0af2800b79f31220e028b75a32907b35c6056f8b036fb9"}
Sep 11 17:18:43.129912 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:43.129882    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-km5mq" event={"ID":"c3c72203-332a-4273-8607-6e7f71377e54","Type":"ContainerStarted","Data":"5c3d7928b1ffbdc6e009c09679f2642f58ce81f599c0563ccff00c32521a68d9"}
Sep 11 17:18:43.130916 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:43.130894    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8jv92" event={"ID":"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd","Type":"ContainerStarted","Data":"a39900ebcc258ffc3febac94a12152f79e448b1a2ae43b2e4a3aadf5671ee9cb"}
Sep 11 17:18:43.131886 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:43.131868    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b2jdx" event={"ID":"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6","Type":"ContainerStarted","Data":"676e4181b41990fd2e8924c299f910fc7b0563015891ed0d229b9f78aaa522de"}
Sep 11 17:18:43.135575 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:43.135032    2562 generic.go:356] "Generic (PLEG): container finished" podID="4d9878da489eb3fb19c741850664c940" containerID="0dd5a8c119dee22e75c95fb82a80482e1a07c6ceab27089a51ea04986eb666f2" exitCode=0
Sep 11 17:18:43.135575 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:43.135285    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-252.ec2.internal" event={"ID":"4d9878da489eb3fb19c741850664c940","Type":"ContainerDied","Data":"0dd5a8c119dee22e75c95fb82a80482e1a07c6ceab27089a51ea04986eb666f2"}
Sep 11 17:18:43.137651 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:43.137633    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-252.ec2.internal" event={"ID":"9202eb18634ea6b37b4f9ec563ddc4eb","Type":"ContainerStarted","Data":"4f76a8b102e4903293c162bdd17828aa2ea8f941f00aca42cc0589475faafe32"}
Sep 11 17:18:43.139045 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:43.139025    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rv2lp" event={"ID":"1050a519-3e79-4de7-acb6-a168148cdbc3","Type":"ContainerStarted","Data":"49bb3146857c83ac2e2487d8fdf90e4595fe7b57052ee8d0f883ff75ca04cbbb"}
Sep 11 17:18:43.140105 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:43.140083    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qjdvg" event={"ID":"edbdd0ae-4406-4088-9230-c68aed929e7b","Type":"ContainerStarted","Data":"1d31f59e49dc55421584b1c7dcd46327281669506464355bee94df48855332e2"}
Sep 11 17:18:43.141208 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:43.141189    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps" event={"ID":"49d0deda-35f9-4165-be19-4c680b5878bd","Type":"ContainerStarted","Data":"a97fbdd4127c9ce5c9220bc37ba9044ec1ed6541fafff93353222ed235e4ebea"}
Sep 11 17:18:43.142193 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:43.142176    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2sn6n" event={"ID":"924a846a-540b-48c3-a5d7-8df3fbaa41e8","Type":"ContainerStarted","Data":"8d1dfa6e422f66b1a6378e99585f8118645b43f91ea2c38a3dd85fa9a5a06606"}
Sep 11 17:18:43.607907 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:43.607279    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a382e1b1-e35e-4975-823b-0e9efe9a84f3-metrics-certs\") pod \"network-metrics-daemon-scdq8\" (UID: \"a382e1b1-e35e-4975-823b-0e9efe9a84f3\") " pod="openshift-multus/network-metrics-daemon-scdq8"
Sep 11 17:18:43.607907 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:43.607357    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85715083-7220-4a0d-9dd0-426a131d8440-original-pull-secret\") pod \"global-pull-secret-syncer-4lmxn\" (UID: \"85715083-7220-4a0d-9dd0-426a131d8440\") " pod="kube-system/global-pull-secret-syncer-4lmxn"
Sep 11 17:18:43.607907 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:43.607478    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:18:43.607907 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:43.607538    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85715083-7220-4a0d-9dd0-426a131d8440-original-pull-secret podName:85715083-7220-4a0d-9dd0-426a131d8440 nodeName:}" failed. No retries permitted until 2026-09-11 17:18:45.60752091 +0000 UTC m=+6.001438344 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/85715083-7220-4a0d-9dd0-426a131d8440-original-pull-secret") pod "global-pull-secret-syncer-4lmxn" (UID: "85715083-7220-4a0d-9dd0-426a131d8440") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:18:43.607907 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:43.607874    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:18:43.608315 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:43.607943    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a382e1b1-e35e-4975-823b-0e9efe9a84f3-metrics-certs podName:a382e1b1-e35e-4975-823b-0e9efe9a84f3 nodeName:}" failed. No retries permitted until 2026-09-11 17:18:45.607925139 +0000 UTC m=+6.001842586 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a382e1b1-e35e-4975-823b-0e9efe9a84f3-metrics-certs") pod "network-metrics-daemon-scdq8" (UID: "a382e1b1-e35e-4975-823b-0e9efe9a84f3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:18:43.809696 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:43.809004    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjvvx\" (UniqueName: \"kubernetes.io/projected/fab8e5a6-345d-4e9d-984e-3019fb93b9a9-kube-api-access-mjvvx\") pod \"network-check-target-frlv9\" (UID: \"fab8e5a6-345d-4e9d-984e-3019fb93b9a9\") " pod="openshift-network-diagnostics/network-check-target-frlv9"
Sep 11 17:18:43.809696 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:43.809196    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:18:43.809696 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:43.809215    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:18:43.809696 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:43.809228    2562 projected.go:196] Error preparing data for projected volume kube-api-access-mjvvx for pod openshift-network-diagnostics/network-check-target-frlv9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:18:43.809696 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:43.809301    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fab8e5a6-345d-4e9d-984e-3019fb93b9a9-kube-api-access-mjvvx podName:fab8e5a6-345d-4e9d-984e-3019fb93b9a9 nodeName:}" failed. No retries permitted until 2026-09-11 17:18:45.809283485 +0000 UTC m=+6.203200939 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mjvvx" (UniqueName: "kubernetes.io/projected/fab8e5a6-345d-4e9d-984e-3019fb93b9a9-kube-api-access-mjvvx") pod "network-check-target-frlv9" (UID: "fab8e5a6-345d-4e9d-984e-3019fb93b9a9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:18:44.035380 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:44.035316    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 17:13:41 +0000 UTC" deadline="2028-03-21 03:33:39.113951216 +0000 UTC"
Sep 11 17:18:44.035380 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:44.035374    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13354h14m55.078581114s"
Sep 11 17:18:44.133805 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:44.133141    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-frlv9"
Sep 11 17:18:44.133805 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:44.133261    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-frlv9" podUID="fab8e5a6-345d-4e9d-984e-3019fb93b9a9"
Sep 11 17:18:44.133805 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:44.133643    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4lmxn"
Sep 11 17:18:44.133805 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:44.133746    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4lmxn" podUID="85715083-7220-4a0d-9dd0-426a131d8440"
Sep 11 17:18:44.178619 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:44.178583    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-252.ec2.internal" event={"ID":"4d9878da489eb3fb19c741850664c940","Type":"ContainerStarted","Data":"08b0d7350de060ff6d92f0dd8c99a6010c36b5abd1e853e6df9cf2a15edb6584"}
Sep 11 17:18:44.197575 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:44.197510    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-252.ec2.internal" podStartSLOduration=3.197489444 podStartE2EDuration="3.197489444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:18:43.167079028 +0000 UTC m=+3.560996484" watchObservedRunningTime="2026-09-11 17:18:44.197489444 +0000 UTC m=+4.591406903"
Sep 11 17:18:44.197879 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:44.197846    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-252.ec2.internal" podStartSLOduration=3.197836466 podStartE2EDuration="3.197836466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:18:44.197141226 +0000 UTC m=+4.591058685" watchObservedRunningTime="2026-09-11 17:18:44.197836466 +0000 UTC m=+4.591753922"
Sep 11 17:18:45.120105 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:45.120017    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-scdq8"
Sep 11 17:18:45.120534 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:45.120163    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-scdq8" podUID="a382e1b1-e35e-4975-823b-0e9efe9a84f3"
Sep 11 17:18:45.624636 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:45.624567    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85715083-7220-4a0d-9dd0-426a131d8440-original-pull-secret\") pod \"global-pull-secret-syncer-4lmxn\" (UID: \"85715083-7220-4a0d-9dd0-426a131d8440\") " pod="kube-system/global-pull-secret-syncer-4lmxn"
Sep 11 17:18:45.624820 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:45.624639    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a382e1b1-e35e-4975-823b-0e9efe9a84f3-metrics-certs\") pod \"network-metrics-daemon-scdq8\" (UID: \"a382e1b1-e35e-4975-823b-0e9efe9a84f3\") " pod="openshift-multus/network-metrics-daemon-scdq8"
Sep 11 17:18:45.624820 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:45.624785    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:18:45.624944 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:45.624850    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a382e1b1-e35e-4975-823b-0e9efe9a84f3-metrics-certs podName:a382e1b1-e35e-4975-823b-0e9efe9a84f3 nodeName:}" failed. No retries permitted until 2026-09-11 17:18:49.624832395 +0000 UTC m=+10.018749843 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a382e1b1-e35e-4975-823b-0e9efe9a84f3-metrics-certs") pod "network-metrics-daemon-scdq8" (UID: "a382e1b1-e35e-4975-823b-0e9efe9a84f3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:18:45.625270 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:45.625252    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:18:45.625359 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:45.625304    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85715083-7220-4a0d-9dd0-426a131d8440-original-pull-secret podName:85715083-7220-4a0d-9dd0-426a131d8440 nodeName:}" failed. No retries permitted until 2026-09-11 17:18:49.625289133 +0000 UTC m=+10.019206568 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/85715083-7220-4a0d-9dd0-426a131d8440-original-pull-secret") pod "global-pull-secret-syncer-4lmxn" (UID: "85715083-7220-4a0d-9dd0-426a131d8440") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:18:45.826353 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:45.826314    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjvvx\" (UniqueName: \"kubernetes.io/projected/fab8e5a6-345d-4e9d-984e-3019fb93b9a9-kube-api-access-mjvvx\") pod \"network-check-target-frlv9\" (UID: \"fab8e5a6-345d-4e9d-984e-3019fb93b9a9\") " pod="openshift-network-diagnostics/network-check-target-frlv9"
Sep 11 17:18:45.826561 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:45.826532    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:18:45.826561 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:45.826555    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:18:45.826692 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:45.826568    2562 projected.go:196] Error preparing data for projected volume kube-api-access-mjvvx for pod openshift-network-diagnostics/network-check-target-frlv9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:18:45.826692 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:45.826625    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fab8e5a6-345d-4e9d-984e-3019fb93b9a9-kube-api-access-mjvvx podName:fab8e5a6-345d-4e9d-984e-3019fb93b9a9 nodeName:}" failed. No retries permitted until 2026-09-11 17:18:49.826607028 +0000 UTC m=+10.220524465 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mjvvx" (UniqueName: "kubernetes.io/projected/fab8e5a6-345d-4e9d-984e-3019fb93b9a9-kube-api-access-mjvvx") pod "network-check-target-frlv9" (UID: "fab8e5a6-345d-4e9d-984e-3019fb93b9a9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:18:46.123370 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:46.123295    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-frlv9"
Sep 11 17:18:46.123802 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:46.123418    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-frlv9" podUID="fab8e5a6-345d-4e9d-984e-3019fb93b9a9"
Sep 11 17:18:46.123802 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:46.123776    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4lmxn"
Sep 11 17:18:46.123917 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:46.123875    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4lmxn" podUID="85715083-7220-4a0d-9dd0-426a131d8440"
Sep 11 17:18:47.119594 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:47.119559    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-scdq8"
Sep 11 17:18:47.119776 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:47.119706    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-scdq8" podUID="a382e1b1-e35e-4975-823b-0e9efe9a84f3"
Sep 11 17:18:48.123796 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:48.123761    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4lmxn"
Sep 11 17:18:48.124225 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:48.123867    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4lmxn" podUID="85715083-7220-4a0d-9dd0-426a131d8440"
Sep 11 17:18:48.124279 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:48.124234    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-frlv9"
Sep 11 17:18:48.124332 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:48.124311    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-frlv9" podUID="fab8e5a6-345d-4e9d-984e-3019fb93b9a9"
Sep 11 17:18:49.119566 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:49.119491    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-scdq8"
Sep 11 17:18:49.119718 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:49.119619    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-scdq8" podUID="a382e1b1-e35e-4975-823b-0e9efe9a84f3"
Sep 11 17:18:49.661503 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:49.661462    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a382e1b1-e35e-4975-823b-0e9efe9a84f3-metrics-certs\") pod \"network-metrics-daemon-scdq8\" (UID: \"a382e1b1-e35e-4975-823b-0e9efe9a84f3\") " pod="openshift-multus/network-metrics-daemon-scdq8"
Sep 11 17:18:49.661957 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:49.661535    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85715083-7220-4a0d-9dd0-426a131d8440-original-pull-secret\") pod \"global-pull-secret-syncer-4lmxn\" (UID: \"85715083-7220-4a0d-9dd0-426a131d8440\") " pod="kube-system/global-pull-secret-syncer-4lmxn"
Sep 11 17:18:49.661957 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:49.661647    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:18:49.661957 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:49.661654    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:18:49.661957 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:49.661699    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85715083-7220-4a0d-9dd0-426a131d8440-original-pull-secret podName:85715083-7220-4a0d-9dd0-426a131d8440 nodeName:}" failed. No retries permitted until 2026-09-11 17:18:57.661685332 +0000 UTC m=+18.055602765 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/85715083-7220-4a0d-9dd0-426a131d8440-original-pull-secret") pod "global-pull-secret-syncer-4lmxn" (UID: "85715083-7220-4a0d-9dd0-426a131d8440") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:18:49.661957 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:49.661731    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a382e1b1-e35e-4975-823b-0e9efe9a84f3-metrics-certs podName:a382e1b1-e35e-4975-823b-0e9efe9a84f3 nodeName:}" failed. No retries permitted until 2026-09-11 17:18:57.661706534 +0000 UTC m=+18.055623986 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a382e1b1-e35e-4975-823b-0e9efe9a84f3-metrics-certs") pod "network-metrics-daemon-scdq8" (UID: "a382e1b1-e35e-4975-823b-0e9efe9a84f3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:18:49.862777 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:49.862738    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjvvx\" (UniqueName: \"kubernetes.io/projected/fab8e5a6-345d-4e9d-984e-3019fb93b9a9-kube-api-access-mjvvx\") pod \"network-check-target-frlv9\" (UID: \"fab8e5a6-345d-4e9d-984e-3019fb93b9a9\") " pod="openshift-network-diagnostics/network-check-target-frlv9"
Sep 11 17:18:49.863091 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:49.862938    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:18:49.863091 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:49.862967    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:18:49.863091 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:49.862981    2562 projected.go:196] Error preparing data for projected volume kube-api-access-mjvvx for pod openshift-network-diagnostics/network-check-target-frlv9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:18:49.863091 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:49.863065    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fab8e5a6-345d-4e9d-984e-3019fb93b9a9-kube-api-access-mjvvx podName:fab8e5a6-345d-4e9d-984e-3019fb93b9a9 nodeName:}" failed. No retries permitted until 2026-09-11 17:18:57.86304348 +0000 UTC m=+18.256960916 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mjvvx" (UniqueName: "kubernetes.io/projected/fab8e5a6-345d-4e9d-984e-3019fb93b9a9-kube-api-access-mjvvx") pod "network-check-target-frlv9" (UID: "fab8e5a6-345d-4e9d-984e-3019fb93b9a9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:18:50.123152 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:50.123078    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4lmxn"
Sep 11 17:18:50.123297 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:50.123192    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4lmxn" podUID="85715083-7220-4a0d-9dd0-426a131d8440"
Sep 11 17:18:50.123583 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:50.123443    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-frlv9"
Sep 11 17:18:50.123583 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:50.123542    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-frlv9" podUID="fab8e5a6-345d-4e9d-984e-3019fb93b9a9"
Sep 11 17:18:51.119354 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:51.119307    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-scdq8"
Sep 11 17:18:51.119770 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:51.119473    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-scdq8" podUID="a382e1b1-e35e-4975-823b-0e9efe9a84f3"
Sep 11 17:18:52.119405 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:52.119372    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-frlv9"
Sep 11 17:18:52.119841 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:52.119500    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-frlv9" podUID="fab8e5a6-345d-4e9d-984e-3019fb93b9a9"
Sep 11 17:18:52.119841 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:52.119549    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4lmxn"
Sep 11 17:18:52.119841 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:52.119652    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4lmxn" podUID="85715083-7220-4a0d-9dd0-426a131d8440"
Sep 11 17:18:53.119163 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:53.119088    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-scdq8"
Sep 11 17:18:53.119298 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:53.119227    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-scdq8" podUID="a382e1b1-e35e-4975-823b-0e9efe9a84f3"
Sep 11 17:18:54.119242 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:54.119207    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-frlv9"
Sep 11 17:18:54.120087 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:54.119319    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-frlv9" podUID="fab8e5a6-345d-4e9d-984e-3019fb93b9a9"
Sep 11 17:18:54.120087 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:54.119377    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4lmxn"
Sep 11 17:18:54.120087 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:54.119460    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4lmxn" podUID="85715083-7220-4a0d-9dd0-426a131d8440"
Sep 11 17:18:55.119855 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:55.119817    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-scdq8"
Sep 11 17:18:55.120368 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:55.119970    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-scdq8" podUID="a382e1b1-e35e-4975-823b-0e9efe9a84f3"
Sep 11 17:18:56.120095 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:56.120057    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4lmxn"
Sep 11 17:18:56.120477 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:56.120068    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-frlv9"
Sep 11 17:18:56.120477 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:56.120190    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4lmxn" podUID="85715083-7220-4a0d-9dd0-426a131d8440"
Sep 11 17:18:56.120477 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:56.120257    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-frlv9" podUID="fab8e5a6-345d-4e9d-984e-3019fb93b9a9"
Sep 11 17:18:57.119556 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:57.119519    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-scdq8"
Sep 11 17:18:57.119746 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:57.119662    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-scdq8" podUID="a382e1b1-e35e-4975-823b-0e9efe9a84f3"
Sep 11 17:18:57.717120 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:57.717082    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a382e1b1-e35e-4975-823b-0e9efe9a84f3-metrics-certs\") pod \"network-metrics-daemon-scdq8\" (UID: \"a382e1b1-e35e-4975-823b-0e9efe9a84f3\") " pod="openshift-multus/network-metrics-daemon-scdq8"
Sep 11 17:18:57.717583 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:57.717162    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85715083-7220-4a0d-9dd0-426a131d8440-original-pull-secret\") pod \"global-pull-secret-syncer-4lmxn\" (UID: \"85715083-7220-4a0d-9dd0-426a131d8440\") " pod="kube-system/global-pull-secret-syncer-4lmxn"
Sep 11 17:18:57.717583 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:57.717214    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:18:57.717583 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:57.717262    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:18:57.717583 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:57.717302    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a382e1b1-e35e-4975-823b-0e9efe9a84f3-metrics-certs podName:a382e1b1-e35e-4975-823b-0e9efe9a84f3 nodeName:}" failed. No retries permitted until 2026-09-11 17:19:13.717279763 +0000 UTC m=+34.111197214 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a382e1b1-e35e-4975-823b-0e9efe9a84f3-metrics-certs") pod "network-metrics-daemon-scdq8" (UID: "a382e1b1-e35e-4975-823b-0e9efe9a84f3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:18:57.717583 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:57.717323    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85715083-7220-4a0d-9dd0-426a131d8440-original-pull-secret podName:85715083-7220-4a0d-9dd0-426a131d8440 nodeName:}" failed. No retries permitted until 2026-09-11 17:19:13.717312727 +0000 UTC m=+34.111230160 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/85715083-7220-4a0d-9dd0-426a131d8440-original-pull-secret") pod "global-pull-secret-syncer-4lmxn" (UID: "85715083-7220-4a0d-9dd0-426a131d8440") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:18:57.918285 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:57.918253    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjvvx\" (UniqueName: \"kubernetes.io/projected/fab8e5a6-345d-4e9d-984e-3019fb93b9a9-kube-api-access-mjvvx\") pod \"network-check-target-frlv9\" (UID: \"fab8e5a6-345d-4e9d-984e-3019fb93b9a9\") " pod="openshift-network-diagnostics/network-check-target-frlv9"
Sep 11 17:18:57.918448 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:57.918389    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:18:57.918448 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:57.918411    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:18:57.918448 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:57.918422    2562 projected.go:196] Error preparing data for projected volume kube-api-access-mjvvx for pod openshift-network-diagnostics/network-check-target-frlv9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:18:57.918592 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:57.918474    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fab8e5a6-345d-4e9d-984e-3019fb93b9a9-kube-api-access-mjvvx podName:fab8e5a6-345d-4e9d-984e-3019fb93b9a9 nodeName:}" failed. No retries permitted until 2026-09-11 17:19:13.918461788 +0000 UTC m=+34.312379221 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mjvvx" (UniqueName: "kubernetes.io/projected/fab8e5a6-345d-4e9d-984e-3019fb93b9a9-kube-api-access-mjvvx") pod "network-check-target-frlv9" (UID: "fab8e5a6-345d-4e9d-984e-3019fb93b9a9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:18:58.120075 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:58.119978    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-frlv9"
Sep 11 17:18:58.120235 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:58.120129    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-frlv9" podUID="fab8e5a6-345d-4e9d-984e-3019fb93b9a9"
Sep 11 17:18:58.120235 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:58.120183    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4lmxn"
Sep 11 17:18:58.120339 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:58.120296    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4lmxn" podUID="85715083-7220-4a0d-9dd0-426a131d8440"
Sep 11 17:18:59.120038 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:18:59.120004    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-scdq8"
Sep 11 17:18:59.120455 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:18:59.120165    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-scdq8" podUID="a382e1b1-e35e-4975-823b-0e9efe9a84f3"
Sep 11 17:19:00.119748 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:00.119712    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4lmxn"
Sep 11 17:19:00.119909 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:00.119781    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-frlv9"
Sep 11 17:19:00.119909 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:00.119794    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4lmxn" podUID="85715083-7220-4a0d-9dd0-426a131d8440"
Sep 11 17:19:00.119909 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:00.119860    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-frlv9" podUID="fab8e5a6-345d-4e9d-984e-3019fb93b9a9"
Sep 11 17:19:01.119605 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:01.119580    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-scdq8"
Sep 11 17:19:01.120071 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:01.119685    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-scdq8" podUID="a382e1b1-e35e-4975-823b-0e9efe9a84f3"
Sep 11 17:19:01.221454 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:01.221417    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2sn6n" event={"ID":"924a846a-540b-48c3-a5d7-8df3fbaa41e8","Type":"ContainerStarted","Data":"a8c5c790c703331cfe6fedb2de6e7831758570c0ddf550c1d8b36f85110d09a1"}
Sep 11 17:19:01.223391 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:01.223360    2562 generic.go:356] "Generic (PLEG): container finished" podID="8aa0128b-3d1f-4a33-9051-b9eb3edc789b" containerID="339585078ae9aa4c58c2465a5717d2b54546b520ace869d1eb3463b9e7d2edf5" exitCode=0
Sep 11 17:19:01.223526 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:01.223454    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wc7rv" event={"ID":"8aa0128b-3d1f-4a33-9051-b9eb3edc789b","Type":"ContainerDied","Data":"339585078ae9aa4c58c2465a5717d2b54546b520ace869d1eb3463b9e7d2edf5"}
Sep 11 17:19:01.225384 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:01.225354    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg9g2" event={"ID":"4cf09e6c-e060-4f10-a0e8-03f505f8e72f","Type":"ContainerStarted","Data":"8544026dd9f3bd1b92142c7bc2624e8d7f9d722709bcdc6ed41bb44b70f6d364"}
Sep 11 17:19:01.227044 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:01.227022    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-km5mq" event={"ID":"c3c72203-332a-4273-8607-6e7f71377e54","Type":"ContainerStarted","Data":"a727b61866a16acb6ff49e2052abfa12d5456845b183e2db65ac4a73b04f3789"}
Sep 11 17:19:01.228518 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:01.228497    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8jv92" event={"ID":"22a4137e-aa6e-4cbd-bd97-8e252cfb41bd","Type":"ContainerStarted","Data":"2368d2f9e93ccc3c2ffaa6bc42e63967243454115ffddc148b586e2ec60fedca"}
Sep 11 17:19:01.230042 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:01.230023    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b2jdx" event={"ID":"bd7d3816-e8d5-4f64-8c24-4ea88a02f4d6","Type":"ContainerStarted","Data":"c9f894a0f952a812bbdf5cf3b294d8a0f80109aa3bac7c9728abf40e66cd0c41"}
Sep 11 17:19:01.231350 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:01.231326    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qjdvg" event={"ID":"edbdd0ae-4406-4088-9230-c68aed929e7b","Type":"ContainerStarted","Data":"1df04d07b6347df9a8a09a92a4e120b81596e301ccd511b7df33d95f2bc47f15"}
Sep 11 17:19:01.234132 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:01.234112    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps" event={"ID":"49d0deda-35f9-4165-be19-4c680b5878bd","Type":"ContainerStarted","Data":"71ef53f5039943e9bb9bf0a9c72ac6df305dd91dd7329a2fe21755ee4db12c6b"}
Sep 11 17:19:01.234211 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:01.234168    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps" event={"ID":"49d0deda-35f9-4165-be19-4c680b5878bd","Type":"ContainerStarted","Data":"dc9142e05ffec71eec38ed361cd67148a929cf9e26a1b2f1349ab5766ff1140b"}
Sep 11 17:19:01.234211 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:01.234198    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps" event={"ID":"49d0deda-35f9-4165-be19-4c680b5878bd","Type":"ContainerStarted","Data":"532b319ff411ac89493d5385929ec887b25e7de2076c771c96527d26cc3f6c5d"}
Sep 11 17:19:01.234272 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:01.234227    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps" event={"ID":"49d0deda-35f9-4165-be19-4c680b5878bd","Type":"ContainerStarted","Data":"c191a294bcc6fdcf8490b205d8e5ef931c4800b252cd75fe10316b22baabbf7c"}
Sep 11 17:19:01.234272 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:01.234258    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps" event={"ID":"49d0deda-35f9-4165-be19-4c680b5878bd","Type":"ContainerStarted","Data":"0a2aeec319e516ec754b9a1363c5ece1d654622ac54e364fd1e928852fab7bd1"}
Sep 11 17:19:01.234326 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:01.234284    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps" event={"ID":"49d0deda-35f9-4165-be19-4c680b5878bd","Type":"ContainerStarted","Data":"1386dc0c0967d9427b44c33e420824d16f17625c1889168d17929f9fc5eca839"}
Sep 11 17:19:01.252331 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:01.252287    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qjdvg" podStartSLOduration=8.692193349 podStartE2EDuration="22.252274766s" totalImagesPullingTime="13.560081417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:18:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:01.252087775 +0000 UTC m=+21.646005230" watchObservedRunningTime="2026-09-11 17:19:01.252274766 +0000 UTC m=+21.646192221"
Sep 11 17:19:01.252657 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:01.252638    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2sn6n" podStartSLOduration=7.698498938 podStartE2EDuration="21.252632671s" totalImagesPullingTime="13.554133733s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:18:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:01.23688664 +0000 UTC m=+21.630804095" watchObservedRunningTime="2026-09-11 17:19:01.252632671 +0000 UTC m=+21.646550125"
Sep 11 17:19:01.286935 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:01.286895    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-b2jdx" podStartSLOduration=4.840302873 podStartE2EDuration="22.286880828s" totalImagesPullingTime="17.446577955s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:18:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:01.268707404 +0000 UTC m=+21.662624860" watchObservedRunningTime="2026-09-11 17:19:01.286880828 +0000 UTC m=+21.680798282"
Sep 11 17:19:01.287271 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:01.287251    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-km5mq" podStartSLOduration=4.845820577 podStartE2EDuration="22.287246052s" totalImagesPullingTime="17.441425475s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:18:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:01.286781526 +0000 UTC m=+21.680698980" watchObservedRunningTime="2026-09-11 17:19:01.287246052 +0000 UTC m=+21.681163508"
Sep 11 17:19:01.310248 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:01.310208    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8jv92" podStartSLOduration=3.611978929 podStartE2EDuration="21.31019454s" totalImagesPullingTime="17.698215611s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:18:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:01.310041457 +0000 UTC m=+21.703958912" watchObservedRunningTime="2026-09-11 17:19:01.31019454 +0000 UTC m=+21.704111994"
Sep 11 17:19:01.593970 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:01.593943    2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 17:19:02.043172 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:02.043014    2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T17:19:01.593966908Z","UUID":"80509d1f-f3ff-453b-a0d9-5f7d075367c7","Handler":null,"Name":"","Endpoint":""}
Sep 11 17:19:02.045062 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:02.045038    2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 17:19:02.045062 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:02.045069    2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 17:19:02.119690 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:02.119657    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-frlv9"
Sep 11 17:19:02.120256 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:02.119668    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4lmxn"
Sep 11 17:19:02.120256 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:02.119773    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-frlv9" podUID="fab8e5a6-345d-4e9d-984e-3019fb93b9a9"
Sep 11 17:19:02.120256 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:02.119889    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4lmxn" podUID="85715083-7220-4a0d-9dd0-426a131d8440"
Sep 11 17:19:02.238832 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:02.238790    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg9g2" event={"ID":"4cf09e6c-e060-4f10-a0e8-03f505f8e72f","Type":"ContainerStarted","Data":"51569ba499580dc30e02d0d88fbde012c103647a83f1b8bf896fc0cd6fedb62d"}
Sep 11 17:19:02.240837 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:02.240798    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rv2lp" event={"ID":"1050a519-3e79-4de7-acb6-a168148cdbc3","Type":"ContainerStarted","Data":"223fabc2f533f65108fa441e0a8cb051da0e9172f426cf5f76c8d247dc25b4cc"}
Sep 11 17:19:02.257159 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:02.256954    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rv2lp" podStartSLOduration=5.817416668 podStartE2EDuration="23.256932835s" totalImagesPullingTime="17.439516167s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:18:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:02.256364833 +0000 UTC m=+22.650282288" watchObservedRunningTime="2026-09-11 17:19:02.256932835 +0000 UTC m=+22.650850291"
Sep 11 17:19:03.119539 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:03.119277    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-scdq8"
Sep 11 17:19:03.119753 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:03.119691    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-scdq8" podUID="a382e1b1-e35e-4975-823b-0e9efe9a84f3"
Sep 11 17:19:03.246010 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:03.245914    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg9g2" event={"ID":"4cf09e6c-e060-4f10-a0e8-03f505f8e72f","Type":"ContainerStarted","Data":"9760ffb99474cb033cabc621d093f1331cba2627e9e8a8caf50ff00746534de3"}
Sep 11 17:19:03.249765 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:03.249742    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps" event={"ID":"49d0deda-35f9-4165-be19-4c680b5878bd","Type":"ContainerStarted","Data":"7a16341f53decf338afa118103b604d0ae80486edf478ce740c70f71b13ff55e"}
Sep 11 17:19:03.262696 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:03.262652    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg9g2" podStartSLOduration=3.752795571 podStartE2EDuration="23.262638456s" totalImagesPullingTime="19.509842885s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:18:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:03.262600227 +0000 UTC m=+23.656517682" watchObservedRunningTime="2026-09-11 17:19:03.262638456 +0000 UTC m=+23.656555912"
Sep 11 17:19:04.119785 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:04.119750    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4lmxn"
Sep 11 17:19:04.120255 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:04.119866    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4lmxn" podUID="85715083-7220-4a0d-9dd0-426a131d8440"
Sep 11 17:19:04.120255 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:04.119926    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-frlv9"
Sep 11 17:19:04.120255 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:04.120034    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-frlv9" podUID="fab8e5a6-345d-4e9d-984e-3019fb93b9a9"
Sep 11 17:19:04.801075 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:04.801039    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qjdvg"
Sep 11 17:19:04.801657 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:04.801637    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qjdvg"
Sep 11 17:19:05.119198 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:05.119167    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-scdq8"
Sep 11 17:19:05.119357 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:05.119320    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-scdq8" podUID="a382e1b1-e35e-4975-823b-0e9efe9a84f3"
Sep 11 17:19:05.254049 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:05.254020    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qjdvg"
Sep 11 17:19:05.254580 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:05.254545    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qjdvg"
Sep 11 17:19:06.119304 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:06.119259    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4lmxn"
Sep 11 17:19:06.119304 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:06.119295    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-frlv9"
Sep 11 17:19:06.119545 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:06.119431    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4lmxn" podUID="85715083-7220-4a0d-9dd0-426a131d8440"
Sep 11 17:19:06.119545 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:06.119520    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-frlv9" podUID="fab8e5a6-345d-4e9d-984e-3019fb93b9a9"
Sep 11 17:19:06.259853 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:06.259815    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps" event={"ID":"49d0deda-35f9-4165-be19-4c680b5878bd","Type":"ContainerStarted","Data":"d0c9eb65b56bb2a4c19dc1f132944e3ab458c4fb8d31c938d5af90adfd273488"}
Sep 11 17:19:07.119968 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:07.119781    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-scdq8"
Sep 11 17:19:07.120147 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:07.120060    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-scdq8" podUID="a382e1b1-e35e-4975-823b-0e9efe9a84f3"
Sep 11 17:19:07.263725 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:07.263693    2562 generic.go:356] "Generic (PLEG): container finished" podID="8aa0128b-3d1f-4a33-9051-b9eb3edc789b" containerID="3a811847c338534ea8434585bfea10339793b98671e92a9834674b64a8ca2411" exitCode=0
Sep 11 17:19:07.264712 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:07.263780    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wc7rv" event={"ID":"8aa0128b-3d1f-4a33-9051-b9eb3edc789b","Type":"ContainerDied","Data":"3a811847c338534ea8434585bfea10339793b98671e92a9834674b64a8ca2411"}
Sep 11 17:19:07.264712 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:07.264066    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:19:07.264712 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:07.264141    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:19:07.264712 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:07.264189    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:19:07.278902 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:07.278875    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:19:07.279187 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:07.278970    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:19:07.329388 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:07.329340    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps" podStartSLOduration=9.843773159 podStartE2EDuration="27.329323671s" totalImagesPullingTime="17.485550512s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:18:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:07.32629303 +0000 UTC m=+27.720210484" watchObservedRunningTime="2026-09-11 17:19:07.329323671 +0000 UTC m=+27.723241129"
Sep 11 17:19:08.119597 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:08.119562    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-frlv9"
Sep 11 17:19:08.119804 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:08.119691    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-frlv9" podUID="fab8e5a6-345d-4e9d-984e-3019fb93b9a9"
Sep 11 17:19:08.119907 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:08.119562    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4lmxn"
Sep 11 17:19:08.120056 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:08.120019    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4lmxn" podUID="85715083-7220-4a0d-9dd0-426a131d8440"
Sep 11 17:19:08.132850 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:08.132822    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4lmxn"]
Sep 11 17:19:08.135550 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:08.135527    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-scdq8"]
Sep 11 17:19:08.135726 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:08.135639    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-scdq8"
Sep 11 17:19:08.135794 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:08.135737    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-scdq8" podUID="a382e1b1-e35e-4975-823b-0e9efe9a84f3"
Sep 11 17:19:08.146445 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:08.146422    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-frlv9"]
Sep 11 17:19:08.266285 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:08.266208    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-frlv9"
Sep 11 17:19:08.266285 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:08.266227    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4lmxn"
Sep 11 17:19:08.267068 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:08.266330    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-frlv9" podUID="fab8e5a6-345d-4e9d-984e-3019fb93b9a9"
Sep 11 17:19:08.267068 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:08.266390    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4lmxn" podUID="85715083-7220-4a0d-9dd0-426a131d8440"
Sep 11 17:19:09.269944 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:09.269907    2562 generic.go:356] "Generic (PLEG): container finished" podID="8aa0128b-3d1f-4a33-9051-b9eb3edc789b" containerID="c564840da603efa669db957b0b1f6e3a8a763c8ac6c4a7fd3dab085b5607d2c3" exitCode=0
Sep 11 17:19:09.270353 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:09.269984    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wc7rv" event={"ID":"8aa0128b-3d1f-4a33-9051-b9eb3edc789b","Type":"ContainerDied","Data":"c564840da603efa669db957b0b1f6e3a8a763c8ac6c4a7fd3dab085b5607d2c3"}
Sep 11 17:19:10.120264 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:10.120233    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-frlv9"
Sep 11 17:19:10.120433 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:10.120316    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-frlv9" podUID="fab8e5a6-345d-4e9d-984e-3019fb93b9a9"
Sep 11 17:19:10.120433 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:10.120411    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-scdq8"
Sep 11 17:19:10.120564 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:10.120543    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-scdq8" podUID="a382e1b1-e35e-4975-823b-0e9efe9a84f3"
Sep 11 17:19:10.120605 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:10.120593    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4lmxn"
Sep 11 17:19:10.120697 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:10.120681    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4lmxn" podUID="85715083-7220-4a0d-9dd0-426a131d8440"
Sep 11 17:19:11.277943 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:11.277910    2562 generic.go:356] "Generic (PLEG): container finished" podID="8aa0128b-3d1f-4a33-9051-b9eb3edc789b" containerID="b8a7e6f7f3067b4f199d35fda8b77cf5d65eb6366b5f0ebf6ef6100c451799e0" exitCode=0
Sep 11 17:19:11.278337 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:11.277972    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wc7rv" event={"ID":"8aa0128b-3d1f-4a33-9051-b9eb3edc789b","Type":"ContainerDied","Data":"b8a7e6f7f3067b4f199d35fda8b77cf5d65eb6366b5f0ebf6ef6100c451799e0"}
Sep 11 17:19:12.119406 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:12.119373    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-frlv9"
Sep 11 17:19:12.119406 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:12.119388    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4lmxn"
Sep 11 17:19:12.119660 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:12.119552    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-frlv9" podUID="fab8e5a6-345d-4e9d-984e-3019fb93b9a9"
Sep 11 17:19:12.119660 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:12.119564    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-scdq8"
Sep 11 17:19:12.119777 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:12.119664    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4lmxn" podUID="85715083-7220-4a0d-9dd0-426a131d8440"
Sep 11 17:19:12.119825 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:12.119773    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-scdq8" podUID="a382e1b1-e35e-4975-823b-0e9efe9a84f3"
Sep 11 17:19:13.308054 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.307917    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-252.ec2.internal" event="NodeReady"
Sep 11 17:19:13.308626 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.308101    2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 17:19:13.389868 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.389834    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d58fbcc7c-65k7r"]
Sep 11 17:19:13.417556 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.417529    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d58fbcc7c-65k7r"]
Sep 11 17:19:13.417720 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.417640    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r"
Sep 11 17:19:13.420899 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.420876    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 17:19:13.421709 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.421685    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-w4wtw\""
Sep 11 17:19:13.423308 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.423285    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 17:19:13.424116 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.424099    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 17:19:13.430251 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.430228    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-z4bx7"]
Sep 11 17:19:13.431574 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.431556    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 17:19:13.455439 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.455414    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gnrll"]
Sep 11 17:19:13.455550 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.455530    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z4bx7"
Sep 11 17:19:13.468896 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.468871    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 17:19:13.469019 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.468878    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bj8s9\""
Sep 11 17:19:13.469689 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.469673    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z4bx7"]
Sep 11 17:19:13.469753 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.469708    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gnrll"]
Sep 11 17:19:13.469753 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.469743    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jd95w"]
Sep 11 17:19:13.469888 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.469873    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gnrll"
Sep 11 17:19:13.470505 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.470487    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 17:19:13.475935 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.475907    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 17:19:13.476075 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.475947    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 17:19:13.476745 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.476727    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p99b8\""
Sep 11 17:19:13.477098 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.477084    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 17:19:13.477370 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.477352    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 17:19:13.485780 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.485760    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jd95w"]
Sep 11 17:19:13.485870 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.485859    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jd95w"
Sep 11 17:19:13.489352 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.489332    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 17:19:13.489459 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.489388    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 17:19:13.490162 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.490138    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wkt9z\""
Sep 11 17:19:13.493569 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.493550    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 17:19:13.526479 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.526460    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-km5mq_c3c72203-332a-4273-8607-6e7f71377e54/dns-node-resolver/0.log"
Sep 11 17:19:13.530773 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.530754    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/025cbfc6-fdb8-41f9-b574-9c6accc6b576-ca-trust-extracted\") pod \"image-registry-5d58fbcc7c-65k7r\" (UID: \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\") " pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r"
Sep 11 17:19:13.530868 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.530786    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/025cbfc6-fdb8-41f9-b574-9c6accc6b576-bound-sa-token\") pod \"image-registry-5d58fbcc7c-65k7r\" (UID: \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\") " pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r"
Sep 11 17:19:13.530868 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.530805    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg6bv\" (UniqueName: \"kubernetes.io/projected/77f356e7-bfb0-47dc-9910-4d49ef7f2d71-kube-api-access-kg6bv\") pod \"dns-default-z4bx7\" (UID: \"77f356e7-bfb0-47dc-9910-4d49ef7f2d71\") " pod="openshift-dns/dns-default-z4bx7"
Sep 11 17:19:13.530868 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.530828    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr4lh\" (UniqueName: \"kubernetes.io/projected/025cbfc6-fdb8-41f9-b574-9c6accc6b576-kube-api-access-vr4lh\") pod \"image-registry-5d58fbcc7c-65k7r\" (UID: \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\") " pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r"
Sep 11 17:19:13.530868 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.530853    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/daabf0e8-8fd4-4a7d-85b5-1713702add55-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnrll\" (UID: \"daabf0e8-8fd4-4a7d-85b5-1713702add55\") " pod="openshift-insights/insights-runtime-extractor-gnrll"
Sep 11 17:19:13.531065 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.530911    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77f356e7-bfb0-47dc-9910-4d49ef7f2d71-tmp-dir\") pod \"dns-default-z4bx7\" (UID: \"77f356e7-bfb0-47dc-9910-4d49ef7f2d71\") " pod="openshift-dns/dns-default-z4bx7"
Sep 11 17:19:13.531065 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.530943    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5rc9\" (UniqueName: \"kubernetes.io/projected/daabf0e8-8fd4-4a7d-85b5-1713702add55-kube-api-access-k5rc9\") pod \"insights-runtime-extractor-gnrll\" (UID: \"daabf0e8-8fd4-4a7d-85b5-1713702add55\") " pod="openshift-insights/insights-runtime-extractor-gnrll"
Sep 11 17:19:13.531065 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.531008    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/025cbfc6-fdb8-41f9-b574-9c6accc6b576-registry-certificates\") pod \"image-registry-5d58fbcc7c-65k7r\" (UID: \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\") " pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r"
Sep 11 17:19:13.531065 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.531026    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/77f356e7-bfb0-47dc-9910-4d49ef7f2d71-config-volume\") pod \"dns-default-z4bx7\" (UID: \"77f356e7-bfb0-47dc-9910-4d49ef7f2d71\") " pod="openshift-dns/dns-default-z4bx7"
Sep 11 17:19:13.531065 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.531044    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/025cbfc6-fdb8-41f9-b574-9c6accc6b576-installation-pull-secrets\") pod \"image-registry-5d58fbcc7c-65k7r\" (UID: \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\") " pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r"
Sep 11 17:19:13.531292 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.531067    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/025cbfc6-fdb8-41f9-b574-9c6accc6b576-image-registry-private-configuration\") pod \"image-registry-5d58fbcc7c-65k7r\" (UID: \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\") " pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r"
Sep 11 17:19:13.531292 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.531100    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/daabf0e8-8fd4-4a7d-85b5-1713702add55-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnrll\" (UID: \"daabf0e8-8fd4-4a7d-85b5-1713702add55\") " pod="openshift-insights/insights-runtime-extractor-gnrll"
Sep 11 17:19:13.531292 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.531152    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77f356e7-bfb0-47dc-9910-4d49ef7f2d71-metrics-tls\") pod \"dns-default-z4bx7\" (UID: \"77f356e7-bfb0-47dc-9910-4d49ef7f2d71\") " pod="openshift-dns/dns-default-z4bx7"
Sep 11 17:19:13.531292 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.531189    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/025cbfc6-fdb8-41f9-b574-9c6accc6b576-registry-tls\") pod \"image-registry-5d58fbcc7c-65k7r\" (UID: \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\") " pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r"
Sep 11 17:19:13.531292 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.531247    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/025cbfc6-fdb8-41f9-b574-9c6accc6b576-trusted-ca\") pod \"image-registry-5d58fbcc7c-65k7r\" (UID: \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\") " pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r"
Sep 11 17:19:13.531292 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.531264    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/daabf0e8-8fd4-4a7d-85b5-1713702add55-crio-socket\") pod \"insights-runtime-extractor-gnrll\" (UID: \"daabf0e8-8fd4-4a7d-85b5-1713702add55\") " pod="openshift-insights/insights-runtime-extractor-gnrll"
Sep 11 17:19:13.531292 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.531283    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/daabf0e8-8fd4-4a7d-85b5-1713702add55-data-volume\") pod \"insights-runtime-extractor-gnrll\" (UID: \"daabf0e8-8fd4-4a7d-85b5-1713702add55\") " pod="openshift-insights/insights-runtime-extractor-gnrll"
Sep 11 17:19:13.632248 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.632215    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/025cbfc6-fdb8-41f9-b574-9c6accc6b576-registry-tls\") pod \"image-registry-5d58fbcc7c-65k7r\" (UID: \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\") " pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r"
Sep 11 17:19:13.632415 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.632283    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/025cbfc6-fdb8-41f9-b574-9c6accc6b576-trusted-ca\") pod \"image-registry-5d58fbcc7c-65k7r\" (UID: \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\") " pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r"
Sep 11 17:19:13.632415 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.632304    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/daabf0e8-8fd4-4a7d-85b5-1713702add55-crio-socket\") pod \"insights-runtime-extractor-gnrll\" (UID: \"daabf0e8-8fd4-4a7d-85b5-1713702add55\") " pod="openshift-insights/insights-runtime-extractor-gnrll"
Sep 11 17:19:13.632415 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.632320    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/daabf0e8-8fd4-4a7d-85b5-1713702add55-data-volume\") pod \"insights-runtime-extractor-gnrll\" (UID: \"daabf0e8-8fd4-4a7d-85b5-1713702add55\") " pod="openshift-insights/insights-runtime-extractor-gnrll"
Sep 11 17:19:13.632415 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.632352    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/025cbfc6-fdb8-41f9-b574-9c6accc6b576-ca-trust-extracted\") pod \"image-registry-5d58fbcc7c-65k7r\" (UID: \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\") " pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r"
Sep 11 17:19:13.632415 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:13.632374    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 17:19:13.632415 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.632390    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/025cbfc6-fdb8-41f9-b574-9c6accc6b576-bound-sa-token\") pod \"image-registry-5d58fbcc7c-65k7r\" (UID: \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\") " pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r"
Sep 11 17:19:13.632415 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:13.632396    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d58fbcc7c-65k7r: secret "image-registry-tls" not found
Sep 11 17:19:13.632415 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.632412    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg6bv\" (UniqueName: \"kubernetes.io/projected/77f356e7-bfb0-47dc-9910-4d49ef7f2d71-kube-api-access-kg6bv\") pod \"dns-default-z4bx7\" (UID: \"77f356e7-bfb0-47dc-9910-4d49ef7f2d71\") " pod="openshift-dns/dns-default-z4bx7"
Sep 11 17:19:13.632824 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:13.632459    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/025cbfc6-fdb8-41f9-b574-9c6accc6b576-registry-tls podName:025cbfc6-fdb8-41f9-b574-9c6accc6b576 nodeName:}" failed. No retries permitted until 2026-09-11 17:19:14.132439454 +0000 UTC m=+34.526356887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/025cbfc6-fdb8-41f9-b574-9c6accc6b576-registry-tls") pod "image-registry-5d58fbcc7c-65k7r" (UID: "025cbfc6-fdb8-41f9-b574-9c6accc6b576") : secret "image-registry-tls" not found
Sep 11 17:19:13.632824 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.632489    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/daabf0e8-8fd4-4a7d-85b5-1713702add55-crio-socket\") pod \"insights-runtime-extractor-gnrll\" (UID: \"daabf0e8-8fd4-4a7d-85b5-1713702add55\") " pod="openshift-insights/insights-runtime-extractor-gnrll"
Sep 11 17:19:13.632824 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.632634    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f87ec688-4995-4c18-9ee4-fe5afda82b21-cert\") pod \"ingress-canary-jd95w\" (UID: \"f87ec688-4995-4c18-9ee4-fe5afda82b21\") " pod="openshift-ingress-canary/ingress-canary-jd95w"
Sep 11 17:19:13.632824 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.632694    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr4lh\" (UniqueName: \"kubernetes.io/projected/025cbfc6-fdb8-41f9-b574-9c6accc6b576-kube-api-access-vr4lh\") pod \"image-registry-5d58fbcc7c-65k7r\" (UID: \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\") " pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r"
Sep 11 17:19:13.632824 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.632704    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/daabf0e8-8fd4-4a7d-85b5-1713702add55-data-volume\") pod \"insights-runtime-extractor-gnrll\" (UID: \"daabf0e8-8fd4-4a7d-85b5-1713702add55\") " pod="openshift-insights/insights-runtime-extractor-gnrll"
Sep 11 17:19:13.632824 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.632738    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/daabf0e8-8fd4-4a7d-85b5-1713702add55-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnrll\" (UID: \"daabf0e8-8fd4-4a7d-85b5-1713702add55\") " pod="openshift-insights/insights-runtime-extractor-gnrll"
Sep 11 17:19:13.632824 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.632777    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77f356e7-bfb0-47dc-9910-4d49ef7f2d71-tmp-dir\") pod \"dns-default-z4bx7\" (UID: \"77f356e7-bfb0-47dc-9910-4d49ef7f2d71\") " pod="openshift-dns/dns-default-z4bx7"
Sep 11 17:19:13.632824 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.632808    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5rc9\" (UniqueName: \"kubernetes.io/projected/daabf0e8-8fd4-4a7d-85b5-1713702add55-kube-api-access-k5rc9\") pod \"insights-runtime-extractor-gnrll\" (UID: \"daabf0e8-8fd4-4a7d-85b5-1713702add55\") " pod="openshift-insights/insights-runtime-extractor-gnrll"
Sep 11 17:19:13.633230 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.632852    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2llz\" (UniqueName: \"kubernetes.io/projected/f87ec688-4995-4c18-9ee4-fe5afda82b21-kube-api-access-j2llz\") pod \"ingress-canary-jd95w\" (UID: \"f87ec688-4995-4c18-9ee4-fe5afda82b21\") " pod="openshift-ingress-canary/ingress-canary-jd95w"
Sep 11 17:19:13.633230 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:13.632865    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 17:19:13.633230 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.632883    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/025cbfc6-fdb8-41f9-b574-9c6accc6b576-registry-certificates\") pod \"image-registry-5d58fbcc7c-65k7r\" (UID: \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\") " pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r"
Sep 11 17:19:13.633230 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:13.632913    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daabf0e8-8fd4-4a7d-85b5-1713702add55-insights-runtime-extractor-tls podName:daabf0e8-8fd4-4a7d-85b5-1713702add55 nodeName:}" failed. No retries permitted until 2026-09-11 17:19:14.13289806 +0000 UTC m=+34.526815517 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/daabf0e8-8fd4-4a7d-85b5-1713702add55-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gnrll" (UID: "daabf0e8-8fd4-4a7d-85b5-1713702add55") : secret "insights-runtime-extractor-tls" not found
Sep 11 17:19:13.633230 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.632930    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/77f356e7-bfb0-47dc-9910-4d49ef7f2d71-config-volume\") pod \"dns-default-z4bx7\" (UID: \"77f356e7-bfb0-47dc-9910-4d49ef7f2d71\") " pod="openshift-dns/dns-default-z4bx7"
Sep 11 17:19:13.633230 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.632946    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/025cbfc6-fdb8-41f9-b574-9c6accc6b576-ca-trust-extracted\") pod \"image-registry-5d58fbcc7c-65k7r\" (UID: \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\") " pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r"
Sep 11 17:19:13.633230 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.632985    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/025cbfc6-fdb8-41f9-b574-9c6accc6b576-installation-pull-secrets\") pod \"image-registry-5d58fbcc7c-65k7r\" (UID: \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\") " pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r"
Sep 11 17:19:13.633230 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.633040    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/025cbfc6-fdb8-41f9-b574-9c6accc6b576-image-registry-private-configuration\") pod \"image-registry-5d58fbcc7c-65k7r\" (UID: \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\") " pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r"
Sep 11 17:19:13.633230 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.633067    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/daabf0e8-8fd4-4a7d-85b5-1713702add55-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnrll\" (UID: \"daabf0e8-8fd4-4a7d-85b5-1713702add55\") " pod="openshift-insights/insights-runtime-extractor-gnrll"
Sep 11 17:19:13.633230 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.633096    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77f356e7-bfb0-47dc-9910-4d49ef7f2d71-metrics-tls\") pod \"dns-default-z4bx7\" (UID: \"77f356e7-bfb0-47dc-9910-4d49ef7f2d71\") " pod="openshift-dns/dns-default-z4bx7"
Sep 11 17:19:13.633230 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.633141    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77f356e7-bfb0-47dc-9910-4d49ef7f2d71-tmp-dir\") pod \"dns-default-z4bx7\" (UID: \"77f356e7-bfb0-47dc-9910-4d49ef7f2d71\") " pod="openshift-dns/dns-default-z4bx7"
Sep 11 17:19:13.633230 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:13.633206    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:19:13.633805 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:13.633250    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77f356e7-bfb0-47dc-9910-4d49ef7f2d71-metrics-tls podName:77f356e7-bfb0-47dc-9910-4d49ef7f2d71 nodeName:}" failed. No retries permitted until 2026-09-11 17:19:14.133236632 +0000 UTC m=+34.527154116 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/77f356e7-bfb0-47dc-9910-4d49ef7f2d71-metrics-tls") pod "dns-default-z4bx7" (UID: "77f356e7-bfb0-47dc-9910-4d49ef7f2d71") : secret "dns-default-metrics-tls" not found
Sep 11 17:19:13.633805 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:13.633206    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 17:19:13.633805 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:13.633282    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/daabf0e8-8fd4-4a7d-85b5-1713702add55-kube-rbac-proxy-cm podName:daabf0e8-8fd4-4a7d-85b5-1713702add55 nodeName:}" failed. No retries permitted until 2026-09-11 17:19:14.133275871 +0000 UTC m=+34.527193304 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/daabf0e8-8fd4-4a7d-85b5-1713702add55-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gnrll" (UID: "daabf0e8-8fd4-4a7d-85b5-1713702add55") : configmap "kube-rbac-proxy" not found
Sep 11 17:19:13.633805 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.633431    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/025cbfc6-fdb8-41f9-b574-9c6accc6b576-registry-certificates\") pod \"image-registry-5d58fbcc7c-65k7r\" (UID: \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\") " pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r"
Sep 11 17:19:13.633805 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.633454    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/77f356e7-bfb0-47dc-9910-4d49ef7f2d71-config-volume\") pod \"dns-default-z4bx7\" (UID: \"77f356e7-bfb0-47dc-9910-4d49ef7f2d71\") " pod="openshift-dns/dns-default-z4bx7"
Sep 11 17:19:13.633805 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.633763    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/025cbfc6-fdb8-41f9-b574-9c6accc6b576-trusted-ca\") pod \"image-registry-5d58fbcc7c-65k7r\" (UID: \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\") " pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r"
Sep 11 17:19:13.637671 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.637531    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/025cbfc6-fdb8-41f9-b574-9c6accc6b576-image-registry-private-configuration\") pod \"image-registry-5d58fbcc7c-65k7r\" (UID: \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\") " pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r"
Sep 11 17:19:13.637813 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.637532    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/025cbfc6-fdb8-41f9-b574-9c6accc6b576-installation-pull-secrets\") pod \"image-registry-5d58fbcc7c-65k7r\" (UID: \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\") " pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r"
Sep 11 17:19:13.648439 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.648413    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/025cbfc6-fdb8-41f9-b574-9c6accc6b576-bound-sa-token\") pod \"image-registry-5d58fbcc7c-65k7r\" (UID: \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\") " pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r"
Sep 11 17:19:13.650253 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.650220    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg6bv\" (UniqueName: \"kubernetes.io/projected/77f356e7-bfb0-47dc-9910-4d49ef7f2d71-kube-api-access-kg6bv\") pod \"dns-default-z4bx7\" (UID: \"77f356e7-bfb0-47dc-9910-4d49ef7f2d71\") " pod="openshift-dns/dns-default-z4bx7"
Sep 11 17:19:13.650583 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.650559    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5rc9\" (UniqueName: \"kubernetes.io/projected/daabf0e8-8fd4-4a7d-85b5-1713702add55-kube-api-access-k5rc9\") pod \"insights-runtime-extractor-gnrll\" (UID: \"daabf0e8-8fd4-4a7d-85b5-1713702add55\") " pod="openshift-insights/insights-runtime-extractor-gnrll"
Sep 11 17:19:13.650699 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.650678    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr4lh\" (UniqueName: \"kubernetes.io/projected/025cbfc6-fdb8-41f9-b574-9c6accc6b576-kube-api-access-vr4lh\") pod \"image-registry-5d58fbcc7c-65k7r\" (UID: \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\") " pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r"
Sep 11 17:19:13.734197 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.734163    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f87ec688-4995-4c18-9ee4-fe5afda82b21-cert\") pod \"ingress-canary-jd95w\" (UID: \"f87ec688-4995-4c18-9ee4-fe5afda82b21\") " pod="openshift-ingress-canary/ingress-canary-jd95w"
Sep 11 17:19:13.734371 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.734220    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85715083-7220-4a0d-9dd0-426a131d8440-original-pull-secret\") pod \"global-pull-secret-syncer-4lmxn\" (UID: \"85715083-7220-4a0d-9dd0-426a131d8440\") " pod="kube-system/global-pull-secret-syncer-4lmxn"
Sep 11 17:19:13.734371 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.734249    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2llz\" (UniqueName: \"kubernetes.io/projected/f87ec688-4995-4c18-9ee4-fe5afda82b21-kube-api-access-j2llz\") pod \"ingress-canary-jd95w\" (UID: \"f87ec688-4995-4c18-9ee4-fe5afda82b21\") " pod="openshift-ingress-canary/ingress-canary-jd95w"
Sep 11 17:19:13.734486 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:13.734399    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:19:13.734486 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:13.734484    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85715083-7220-4a0d-9dd0-426a131d8440-original-pull-secret podName:85715083-7220-4a0d-9dd0-426a131d8440 nodeName:}" failed. No retries permitted until 2026-09-11 17:19:45.734464049 +0000 UTC m=+66.128381501 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/85715083-7220-4a0d-9dd0-426a131d8440-original-pull-secret") pod "global-pull-secret-syncer-4lmxn" (UID: "85715083-7220-4a0d-9dd0-426a131d8440") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:19:13.734604 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.734511    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a382e1b1-e35e-4975-823b-0e9efe9a84f3-metrics-certs\") pod \"network-metrics-daemon-scdq8\" (UID: \"a382e1b1-e35e-4975-823b-0e9efe9a84f3\") " pod="openshift-multus/network-metrics-daemon-scdq8"
Sep 11 17:19:13.734664 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:13.734650    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:19:13.734721 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:13.734706    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a382e1b1-e35e-4975-823b-0e9efe9a84f3-metrics-certs podName:a382e1b1-e35e-4975-823b-0e9efe9a84f3 nodeName:}" failed. No retries permitted until 2026-09-11 17:19:45.734687736 +0000 UTC m=+66.128605173 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a382e1b1-e35e-4975-823b-0e9efe9a84f3-metrics-certs") pod "network-metrics-daemon-scdq8" (UID: "a382e1b1-e35e-4975-823b-0e9efe9a84f3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:19:13.737227 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.737191    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f87ec688-4995-4c18-9ee4-fe5afda82b21-cert\") pod \"ingress-canary-jd95w\" (UID: \"f87ec688-4995-4c18-9ee4-fe5afda82b21\") " pod="openshift-ingress-canary/ingress-canary-jd95w"
Sep 11 17:19:13.744249 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.744226    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2llz\" (UniqueName: \"kubernetes.io/projected/f87ec688-4995-4c18-9ee4-fe5afda82b21-kube-api-access-j2llz\") pod \"ingress-canary-jd95w\" (UID: \"f87ec688-4995-4c18-9ee4-fe5afda82b21\") " pod="openshift-ingress-canary/ingress-canary-jd95w"
Sep 11 17:19:13.794703 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.794675    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jd95w"
Sep 11 17:19:13.936424 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:13.936395    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjvvx\" (UniqueName: \"kubernetes.io/projected/fab8e5a6-345d-4e9d-984e-3019fb93b9a9-kube-api-access-mjvvx\") pod \"network-check-target-frlv9\" (UID: \"fab8e5a6-345d-4e9d-984e-3019fb93b9a9\") " pod="openshift-network-diagnostics/network-check-target-frlv9"
Sep 11 17:19:13.936613 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:13.936590    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:19:13.936658 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:13.936640    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:19:13.936658 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:13.936655    2562 projected.go:196] Error preparing data for projected volume kube-api-access-mjvvx for pod openshift-network-diagnostics/network-check-target-frlv9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:19:13.936789 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:13.936755    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fab8e5a6-345d-4e9d-984e-3019fb93b9a9-kube-api-access-mjvvx podName:fab8e5a6-345d-4e9d-984e-3019fb93b9a9 nodeName:}" failed. No retries permitted until 2026-09-11 17:19:45.936733968 +0000 UTC m=+66.330651406 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mjvvx" (UniqueName: "kubernetes.io/projected/fab8e5a6-345d-4e9d-984e-3019fb93b9a9-kube-api-access-mjvvx") pod "network-check-target-frlv9" (UID: "fab8e5a6-345d-4e9d-984e-3019fb93b9a9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:19:14.071541 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:14.071510    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2sn6n_924a846a-540b-48c3-a5d7-8df3fbaa41e8/node-ca/0.log"
Sep 11 17:19:14.120062 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:14.119980    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-frlv9"
Sep 11 17:19:14.120062 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:14.120032    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4lmxn"
Sep 11 17:19:14.120296 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:14.119980    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-scdq8"
Sep 11 17:19:14.127402 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:14.127379    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 17:19:14.131878 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:14.128350    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 17:19:14.131878 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:14.128366    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 17:19:14.131878 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:14.128373    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 17:19:14.131878 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:14.128841    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vthbj\""
Sep 11 17:19:14.131878 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:14.129277    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gdccs\""
Sep 11 17:19:14.137363 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:14.137125    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/daabf0e8-8fd4-4a7d-85b5-1713702add55-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnrll\" (UID: \"daabf0e8-8fd4-4a7d-85b5-1713702add55\") " pod="openshift-insights/insights-runtime-extractor-gnrll"
Sep 11 17:19:14.137363 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:14.137167    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77f356e7-bfb0-47dc-9910-4d49ef7f2d71-metrics-tls\") pod \"dns-default-z4bx7\" (UID: \"77f356e7-bfb0-47dc-9910-4d49ef7f2d71\") " pod="openshift-dns/dns-default-z4bx7"
Sep 11 17:19:14.137363 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:14.137200    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/025cbfc6-fdb8-41f9-b574-9c6accc6b576-registry-tls\") pod \"image-registry-5d58fbcc7c-65k7r\" (UID: \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\") " pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r"
Sep 11 17:19:14.137363 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:14.137261    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 17:19:14.137363 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:14.137281    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/daabf0e8-8fd4-4a7d-85b5-1713702add55-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnrll\" (UID: \"daabf0e8-8fd4-4a7d-85b5-1713702add55\") " pod="openshift-insights/insights-runtime-extractor-gnrll"
Sep 11 17:19:14.137755 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:14.137368    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/daabf0e8-8fd4-4a7d-85b5-1713702add55-kube-rbac-proxy-cm podName:daabf0e8-8fd4-4a7d-85b5-1713702add55 nodeName:}" failed. No retries permitted until 2026-09-11 17:19:15.137349151 +0000 UTC m=+35.531266598 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/daabf0e8-8fd4-4a7d-85b5-1713702add55-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gnrll" (UID: "daabf0e8-8fd4-4a7d-85b5-1713702add55") : configmap "kube-rbac-proxy" not found
Sep 11 17:19:14.137887 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:14.137444    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:19:14.137887 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:14.137844    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77f356e7-bfb0-47dc-9910-4d49ef7f2d71-metrics-tls podName:77f356e7-bfb0-47dc-9910-4d49ef7f2d71 nodeName:}" failed. No retries permitted until 2026-09-11 17:19:15.137825428 +0000 UTC m=+35.531742877 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/77f356e7-bfb0-47dc-9910-4d49ef7f2d71-metrics-tls") pod "dns-default-z4bx7" (UID: "77f356e7-bfb0-47dc-9910-4d49ef7f2d71") : secret "dns-default-metrics-tls" not found
Sep 11 17:19:14.137887 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:14.137509    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 17:19:14.137887 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:14.137888    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daabf0e8-8fd4-4a7d-85b5-1713702add55-insights-runtime-extractor-tls podName:daabf0e8-8fd4-4a7d-85b5-1713702add55 nodeName:}" failed. No retries permitted until 2026-09-11 17:19:15.137877544 +0000 UTC m=+35.531794979 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/daabf0e8-8fd4-4a7d-85b5-1713702add55-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gnrll" (UID: "daabf0e8-8fd4-4a7d-85b5-1713702add55") : secret "insights-runtime-extractor-tls" not found
Sep 11 17:19:14.138119 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:14.137539    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 17:19:14.138119 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:14.137905    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d58fbcc7c-65k7r: secret "image-registry-tls" not found
Sep 11 17:19:14.138119 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:14.137934    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/025cbfc6-fdb8-41f9-b574-9c6accc6b576-registry-tls podName:025cbfc6-fdb8-41f9-b574-9c6accc6b576 nodeName:}" failed. No retries permitted until 2026-09-11 17:19:15.137925431 +0000 UTC m=+35.531842868 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/025cbfc6-fdb8-41f9-b574-9c6accc6b576-registry-tls") pod "image-registry-5d58fbcc7c-65k7r" (UID: "025cbfc6-fdb8-41f9-b574-9c6accc6b576") : secret "image-registry-tls" not found
Sep 11 17:19:14.142880 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:14.142671    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jd95w"]
Sep 11 17:19:14.146079 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:19:14.146055    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf87ec688_4995_4c18_9ee4_fe5afda82b21.slice/crio-9a98b0f5c04b4cd577381261f6c73024e96827d4092be927f8711248b78dcd62 WatchSource:0}: Error finding container 9a98b0f5c04b4cd577381261f6c73024e96827d4092be927f8711248b78dcd62: Status 404 returned error can't find the container with id 9a98b0f5c04b4cd577381261f6c73024e96827d4092be927f8711248b78dcd62
Sep 11 17:19:14.286065 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:14.286026    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jd95w" event={"ID":"f87ec688-4995-4c18-9ee4-fe5afda82b21","Type":"ContainerStarted","Data":"9a98b0f5c04b4cd577381261f6c73024e96827d4092be927f8711248b78dcd62"}
Sep 11 17:19:14.571479 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:14.571387    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-c6xkt"]
Sep 11 17:19:14.596556 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:14.596528    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-c6xkt"]
Sep 11 17:19:14.596723 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:14.596645    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-c6xkt"
Sep 11 17:19:14.600034 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:14.599980    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 11 17:19:14.600034 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:14.600010    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 11 17:19:14.600971 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:14.600679    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 11 17:19:14.600971 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:14.600713    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-rd9dd\""
Sep 11 17:19:14.600971 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:14.600796    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 11 17:19:14.742964 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:14.742912    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e7643a39-5fc7-4cdc-9b87-507b94e55db3-signing-key\") pod \"service-ca-568869fdc5-c6xkt\" (UID: \"e7643a39-5fc7-4cdc-9b87-507b94e55db3\") " pod="openshift-service-ca/service-ca-568869fdc5-c6xkt"
Sep 11 17:19:14.743161 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:14.742981    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e7643a39-5fc7-4cdc-9b87-507b94e55db3-signing-cabundle\") pod \"service-ca-568869fdc5-c6xkt\" (UID: \"e7643a39-5fc7-4cdc-9b87-507b94e55db3\") " pod="openshift-service-ca/service-ca-568869fdc5-c6xkt"
Sep 11 17:19:14.743161 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:14.743031    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45q5h\" (UniqueName: \"kubernetes.io/projected/e7643a39-5fc7-4cdc-9b87-507b94e55db3-kube-api-access-45q5h\") pod \"service-ca-568869fdc5-c6xkt\" (UID: \"e7643a39-5fc7-4cdc-9b87-507b94e55db3\") " pod="openshift-service-ca/service-ca-568869fdc5-c6xkt"
Sep 11 17:19:14.844461 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:14.844432    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e7643a39-5fc7-4cdc-9b87-507b94e55db3-signing-key\") pod \"service-ca-568869fdc5-c6xkt\" (UID: \"e7643a39-5fc7-4cdc-9b87-507b94e55db3\") " pod="openshift-service-ca/service-ca-568869fdc5-c6xkt"
Sep 11 17:19:14.844640 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:14.844475    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e7643a39-5fc7-4cdc-9b87-507b94e55db3-signing-cabundle\") pod \"service-ca-568869fdc5-c6xkt\" (UID: \"e7643a39-5fc7-4cdc-9b87-507b94e55db3\") " pod="openshift-service-ca/service-ca-568869fdc5-c6xkt"
Sep 11 17:19:14.844640 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:14.844501    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45q5h\" (UniqueName: \"kubernetes.io/projected/e7643a39-5fc7-4cdc-9b87-507b94e55db3-kube-api-access-45q5h\") pod \"service-ca-568869fdc5-c6xkt\" (UID: \"e7643a39-5fc7-4cdc-9b87-507b94e55db3\") " pod="openshift-service-ca/service-ca-568869fdc5-c6xkt"
Sep 11 17:19:14.845344 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:14.845316    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e7643a39-5fc7-4cdc-9b87-507b94e55db3-signing-cabundle\") pod \"service-ca-568869fdc5-c6xkt\" (UID: \"e7643a39-5fc7-4cdc-9b87-507b94e55db3\") " pod="openshift-service-ca/service-ca-568869fdc5-c6xkt"
Sep 11 17:19:14.847607 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:14.847583    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e7643a39-5fc7-4cdc-9b87-507b94e55db3-signing-key\") pod \"service-ca-568869fdc5-c6xkt\" (UID: \"e7643a39-5fc7-4cdc-9b87-507b94e55db3\") " pod="openshift-service-ca/service-ca-568869fdc5-c6xkt"
Sep 11 17:19:14.855427 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:14.855405    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45q5h\" (UniqueName: \"kubernetes.io/projected/e7643a39-5fc7-4cdc-9b87-507b94e55db3-kube-api-access-45q5h\") pod \"service-ca-568869fdc5-c6xkt\" (UID: \"e7643a39-5fc7-4cdc-9b87-507b94e55db3\") " pod="openshift-service-ca/service-ca-568869fdc5-c6xkt"
Sep 11 17:19:14.907432 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:14.907404    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-c6xkt"
Sep 11 17:19:15.147769 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:15.147685    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/daabf0e8-8fd4-4a7d-85b5-1713702add55-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnrll\" (UID: \"daabf0e8-8fd4-4a7d-85b5-1713702add55\") " pod="openshift-insights/insights-runtime-extractor-gnrll"
Sep 11 17:19:15.147769 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:15.147738    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77f356e7-bfb0-47dc-9910-4d49ef7f2d71-metrics-tls\") pod \"dns-default-z4bx7\" (UID: \"77f356e7-bfb0-47dc-9910-4d49ef7f2d71\") " pod="openshift-dns/dns-default-z4bx7"
Sep 11 17:19:15.147769 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:15.147771    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/025cbfc6-fdb8-41f9-b574-9c6accc6b576-registry-tls\") pod \"image-registry-5d58fbcc7c-65k7r\" (UID: \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\") " pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r"
Sep 11 17:19:15.148034 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:15.147816    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 17:19:15.148034 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:15.147849    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/daabf0e8-8fd4-4a7d-85b5-1713702add55-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnrll\" (UID: \"daabf0e8-8fd4-4a7d-85b5-1713702add55\") " pod="openshift-insights/insights-runtime-extractor-gnrll"
Sep 11 17:19:15.148034 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:15.147889    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/daabf0e8-8fd4-4a7d-85b5-1713702add55-kube-rbac-proxy-cm podName:daabf0e8-8fd4-4a7d-85b5-1713702add55 nodeName:}" failed. No retries permitted until 2026-09-11 17:19:17.147869401 +0000 UTC m=+37.541786835 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/daabf0e8-8fd4-4a7d-85b5-1713702add55-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gnrll" (UID: "daabf0e8-8fd4-4a7d-85b5-1713702add55") : configmap "kube-rbac-proxy" not found
Sep 11 17:19:15.148034 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:15.147920    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:19:15.148034 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:15.147971    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 17:19:15.148034 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:15.147982    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77f356e7-bfb0-47dc-9910-4d49ef7f2d71-metrics-tls podName:77f356e7-bfb0-47dc-9910-4d49ef7f2d71 nodeName:}" failed. No retries permitted until 2026-09-11 17:19:17.147965965 +0000 UTC m=+37.541883412 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/77f356e7-bfb0-47dc-9910-4d49ef7f2d71-metrics-tls") pod "dns-default-z4bx7" (UID: "77f356e7-bfb0-47dc-9910-4d49ef7f2d71") : secret "dns-default-metrics-tls" not found
Sep 11 17:19:15.148034 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:15.147986    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d58fbcc7c-65k7r: secret "image-registry-tls" not found
Sep 11 17:19:15.148034 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:15.147929    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 17:19:15.148328 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:15.148044    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/025cbfc6-fdb8-41f9-b574-9c6accc6b576-registry-tls podName:025cbfc6-fdb8-41f9-b574-9c6accc6b576 nodeName:}" failed. No retries permitted until 2026-09-11 17:19:17.148031948 +0000 UTC m=+37.541949385 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/025cbfc6-fdb8-41f9-b574-9c6accc6b576-registry-tls") pod "image-registry-5d58fbcc7c-65k7r" (UID: "025cbfc6-fdb8-41f9-b574-9c6accc6b576") : secret "image-registry-tls" not found
Sep 11 17:19:15.148328 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:15.148058    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daabf0e8-8fd4-4a7d-85b5-1713702add55-insights-runtime-extractor-tls podName:daabf0e8-8fd4-4a7d-85b5-1713702add55 nodeName:}" failed. No retries permitted until 2026-09-11 17:19:17.14805176 +0000 UTC m=+37.541969192 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/daabf0e8-8fd4-4a7d-85b5-1713702add55-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gnrll" (UID: "daabf0e8-8fd4-4a7d-85b5-1713702add55") : secret "insights-runtime-extractor-tls" not found
Sep 11 17:19:16.282148 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:16.282060    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-c6xkt"]
Sep 11 17:19:16.315974 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:19:16.315942    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7643a39_5fc7_4cdc_9b87_507b94e55db3.slice/crio-7e9ce109892ab742aa2b2633a88a6a129a891ec266083d575dd7978bac72a130 WatchSource:0}: Error finding container 7e9ce109892ab742aa2b2633a88a6a129a891ec266083d575dd7978bac72a130: Status 404 returned error can't find the container with id 7e9ce109892ab742aa2b2633a88a6a129a891ec266083d575dd7978bac72a130
Sep 11 17:19:17.165717 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:17.165668    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/025cbfc6-fdb8-41f9-b574-9c6accc6b576-registry-tls\") pod \"image-registry-5d58fbcc7c-65k7r\" (UID: \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\") " pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r"
Sep 11 17:19:17.165888 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:17.165814    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/daabf0e8-8fd4-4a7d-85b5-1713702add55-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnrll\" (UID: \"daabf0e8-8fd4-4a7d-85b5-1713702add55\") " pod="openshift-insights/insights-runtime-extractor-gnrll"
Sep 11 17:19:17.165888 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:17.165837    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 17:19:17.165888 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:17.165873    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d58fbcc7c-65k7r: secret "image-registry-tls" not found
Sep 11 17:19:17.166076 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:17.165874    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/daabf0e8-8fd4-4a7d-85b5-1713702add55-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnrll\" (UID: \"daabf0e8-8fd4-4a7d-85b5-1713702add55\") " pod="openshift-insights/insights-runtime-extractor-gnrll"
Sep 11 17:19:17.166076 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:17.165924    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77f356e7-bfb0-47dc-9910-4d49ef7f2d71-metrics-tls\") pod \"dns-default-z4bx7\" (UID: \"77f356e7-bfb0-47dc-9910-4d49ef7f2d71\") " pod="openshift-dns/dns-default-z4bx7"
Sep 11 17:19:17.166076 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:17.165940    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/025cbfc6-fdb8-41f9-b574-9c6accc6b576-registry-tls podName:025cbfc6-fdb8-41f9-b574-9c6accc6b576 nodeName:}" failed. No retries permitted until 2026-09-11 17:19:21.165918602 +0000 UTC m=+41.559836038 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/025cbfc6-fdb8-41f9-b574-9c6accc6b576-registry-tls") pod "image-registry-5d58fbcc7c-65k7r" (UID: "025cbfc6-fdb8-41f9-b574-9c6accc6b576") : secret "image-registry-tls" not found
Sep 11 17:19:17.166076 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:17.166024    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:19:17.166076 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:17.166069    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77f356e7-bfb0-47dc-9910-4d49ef7f2d71-metrics-tls podName:77f356e7-bfb0-47dc-9910-4d49ef7f2d71 nodeName:}" failed. No retries permitted until 2026-09-11 17:19:21.166058822 +0000 UTC m=+41.559976254 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/77f356e7-bfb0-47dc-9910-4d49ef7f2d71-metrics-tls") pod "dns-default-z4bx7" (UID: "77f356e7-bfb0-47dc-9910-4d49ef7f2d71") : secret "dns-default-metrics-tls" not found
Sep 11 17:19:17.166285 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:17.166119    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 17:19:17.166285 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:17.166148    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daabf0e8-8fd4-4a7d-85b5-1713702add55-insights-runtime-extractor-tls podName:daabf0e8-8fd4-4a7d-85b5-1713702add55 nodeName:}" failed. No retries permitted until 2026-09-11 17:19:21.166138438 +0000 UTC m=+41.560055874 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/daabf0e8-8fd4-4a7d-85b5-1713702add55-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gnrll" (UID: "daabf0e8-8fd4-4a7d-85b5-1713702add55") : secret "insights-runtime-extractor-tls" not found
Sep 11 17:19:17.166695 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:17.166669    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/daabf0e8-8fd4-4a7d-85b5-1713702add55-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnrll\" (UID: \"daabf0e8-8fd4-4a7d-85b5-1713702add55\") " pod="openshift-insights/insights-runtime-extractor-gnrll"
Sep 11 17:19:17.294978 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:17.294940    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-c6xkt" event={"ID":"e7643a39-5fc7-4cdc-9b87-507b94e55db3","Type":"ContainerStarted","Data":"7e9ce109892ab742aa2b2633a88a6a129a891ec266083d575dd7978bac72a130"}
Sep 11 17:19:17.299664 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:17.299632    2562 generic.go:356] "Generic (PLEG): container finished" podID="8aa0128b-3d1f-4a33-9051-b9eb3edc789b" containerID="2f403d5963d5fc17a947c35b9b3b7a2702f64e82d18df6e2a2c123331b5068c4" exitCode=0
Sep 11 17:19:17.299664 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:17.299673    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wc7rv" event={"ID":"8aa0128b-3d1f-4a33-9051-b9eb3edc789b","Type":"ContainerDied","Data":"2f403d5963d5fc17a947c35b9b3b7a2702f64e82d18df6e2a2c123331b5068c4"}
Sep 11 17:19:18.304860 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:18.304826    2562 generic.go:356] "Generic (PLEG): container finished" podID="8aa0128b-3d1f-4a33-9051-b9eb3edc789b" containerID="fd5440b1b83a4a4aae1c17aed804082fc420e118c73d2d687d6eddd03fedf709" exitCode=0
Sep 11 17:19:18.305242 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:18.304867    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wc7rv" event={"ID":"8aa0128b-3d1f-4a33-9051-b9eb3edc789b","Type":"ContainerDied","Data":"fd5440b1b83a4a4aae1c17aed804082fc420e118c73d2d687d6eddd03fedf709"}
Sep 11 17:19:20.312457 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:20.312415    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wc7rv" event={"ID":"8aa0128b-3d1f-4a33-9051-b9eb3edc789b","Type":"ContainerStarted","Data":"a2afdfe8ea5a330b3917e3692e3a0e94459de91b53b8dda83e2b60b87e98272e"}
Sep 11 17:19:20.314023 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:20.313974    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jd95w" event={"ID":"f87ec688-4995-4c18-9ee4-fe5afda82b21","Type":"ContainerStarted","Data":"a3d52347b925f210fd06d8eae1ad08567ab7b968c2e2bfad67138069b22bacd4"}
Sep 11 17:19:20.315414 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:20.315392    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-c6xkt" event={"ID":"e7643a39-5fc7-4cdc-9b87-507b94e55db3","Type":"ContainerStarted","Data":"74a156c3ebcf9e4c3860945159af8f418c5fe99d27fc3607cd724c0900531f5e"}
Sep 11 17:19:20.342563 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:20.342513    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wc7rv" podStartSLOduration=10.405694635 podStartE2EDuration="40.342498835s" totalImagesPullingTime="29.9368042s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:18:40 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:20.342014997 +0000 UTC m=+40.735932451" watchObservedRunningTime="2026-09-11 17:19:20.342498835 +0000 UTC m=+40.736416289"
Sep 11 17:19:20.363861 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:20.363809    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jd95w" podStartSLOduration=2.26640072 podStartE2EDuration="7.36379454s" totalImagesPullingTime="5.09739382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:19:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:20.362252438 +0000 UTC m=+40.756169894" watchObservedRunningTime="2026-09-11 17:19:20.36379454 +0000 UTC m=+40.757711994"
Sep 11 17:19:20.390918 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:20.390854    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-c6xkt" podStartSLOduration=3.470467483 podStartE2EDuration="6.390835624s" totalImagesPullingTime="2.920368141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:19:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:20.39051501 +0000 UTC m=+40.784432465" watchObservedRunningTime="2026-09-11 17:19:20.390835624 +0000 UTC m=+40.784753079"
Sep 11 17:19:21.203057 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:21.203018    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77f356e7-bfb0-47dc-9910-4d49ef7f2d71-metrics-tls\") pod \"dns-default-z4bx7\" (UID: \"77f356e7-bfb0-47dc-9910-4d49ef7f2d71\") " pod="openshift-dns/dns-default-z4bx7"
Sep 11 17:19:21.203235 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:21.203066    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/025cbfc6-fdb8-41f9-b574-9c6accc6b576-registry-tls\") pod \"image-registry-5d58fbcc7c-65k7r\" (UID: \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\") " pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r"
Sep 11 17:19:21.203235 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:21.203189    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:19:21.203235 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:21.203209    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/daabf0e8-8fd4-4a7d-85b5-1713702add55-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnrll\" (UID: \"daabf0e8-8fd4-4a7d-85b5-1713702add55\") " pod="openshift-insights/insights-runtime-extractor-gnrll"
Sep 11 17:19:21.203373 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:21.203257    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77f356e7-bfb0-47dc-9910-4d49ef7f2d71-metrics-tls podName:77f356e7-bfb0-47dc-9910-4d49ef7f2d71 nodeName:}" failed. No retries permitted until 2026-09-11 17:19:29.203243089 +0000 UTC m=+49.597160521 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/77f356e7-bfb0-47dc-9910-4d49ef7f2d71-metrics-tls") pod "dns-default-z4bx7" (UID: "77f356e7-bfb0-47dc-9910-4d49ef7f2d71") : secret "dns-default-metrics-tls" not found
Sep 11 17:19:21.203373 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:21.203286    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 17:19:21.203373 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:21.203193    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 17:19:21.203373 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:21.203315    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d58fbcc7c-65k7r: secret "image-registry-tls" not found
Sep 11 17:19:21.203373 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:21.203324    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daabf0e8-8fd4-4a7d-85b5-1713702add55-insights-runtime-extractor-tls podName:daabf0e8-8fd4-4a7d-85b5-1713702add55 nodeName:}" failed. No retries permitted until 2026-09-11 17:19:29.203312904 +0000 UTC m=+49.597230338 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/daabf0e8-8fd4-4a7d-85b5-1713702add55-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gnrll" (UID: "daabf0e8-8fd4-4a7d-85b5-1713702add55") : secret "insights-runtime-extractor-tls" not found
Sep 11 17:19:21.203373 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:21.203365    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/025cbfc6-fdb8-41f9-b574-9c6accc6b576-registry-tls podName:025cbfc6-fdb8-41f9-b574-9c6accc6b576 nodeName:}" failed. No retries permitted until 2026-09-11 17:19:29.203350016 +0000 UTC m=+49.597267463 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/025cbfc6-fdb8-41f9-b574-9c6accc6b576-registry-tls") pod "image-registry-5d58fbcc7c-65k7r" (UID: "025cbfc6-fdb8-41f9-b574-9c6accc6b576") : secret "image-registry-tls" not found
Sep 11 17:19:29.267005 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:29.266957    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77f356e7-bfb0-47dc-9910-4d49ef7f2d71-metrics-tls\") pod \"dns-default-z4bx7\" (UID: \"77f356e7-bfb0-47dc-9910-4d49ef7f2d71\") " pod="openshift-dns/dns-default-z4bx7"
Sep 11 17:19:29.267421 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:29.267013    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/025cbfc6-fdb8-41f9-b574-9c6accc6b576-registry-tls\") pod \"image-registry-5d58fbcc7c-65k7r\" (UID: \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\") " pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r"
Sep 11 17:19:29.267421 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:29.267071    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/daabf0e8-8fd4-4a7d-85b5-1713702add55-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnrll\" (UID: \"daabf0e8-8fd4-4a7d-85b5-1713702add55\") " pod="openshift-insights/insights-runtime-extractor-gnrll"
Sep 11 17:19:29.269971 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:29.269943    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/daabf0e8-8fd4-4a7d-85b5-1713702add55-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnrll\" (UID: \"daabf0e8-8fd4-4a7d-85b5-1713702add55\") " pod="openshift-insights/insights-runtime-extractor-gnrll"
Sep 11 17:19:29.270100 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:29.270016    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77f356e7-bfb0-47dc-9910-4d49ef7f2d71-metrics-tls\") pod \"dns-default-z4bx7\" (UID: \"77f356e7-bfb0-47dc-9910-4d49ef7f2d71\") " pod="openshift-dns/dns-default-z4bx7"
Sep 11 17:19:29.270100 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:29.270037    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/025cbfc6-fdb8-41f9-b574-9c6accc6b576-registry-tls\") pod \"image-registry-5d58fbcc7c-65k7r\" (UID: \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\") " pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r"
Sep 11 17:19:29.328032 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:29.327988    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r"
Sep 11 17:19:29.365449 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:29.365416    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z4bx7"
Sep 11 17:19:29.379775 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:29.379729    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gnrll"
Sep 11 17:19:29.621066 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:29.620869    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d58fbcc7c-65k7r"]
Sep 11 17:19:29.623802 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:19:29.623767    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod025cbfc6_fdb8_41f9_b574_9c6accc6b576.slice/crio-bc02ba25433da35a57c559c56f9f29d7d5435ce3337adb6ff63d692504ed581e WatchSource:0}: Error finding container bc02ba25433da35a57c559c56f9f29d7d5435ce3337adb6ff63d692504ed581e: Status 404 returned error can't find the container with id bc02ba25433da35a57c559c56f9f29d7d5435ce3337adb6ff63d692504ed581e
Sep 11 17:19:29.668331 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:29.668304    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z4bx7"]
Sep 11 17:19:29.671497 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:19:29.671470    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77f356e7_bfb0_47dc_9910_4d49ef7f2d71.slice/crio-220054fcd716bc0c8fae2c02b34fae40f8fdfbeb3f904b37d4dc52e3db52f118 WatchSource:0}: Error finding container 220054fcd716bc0c8fae2c02b34fae40f8fdfbeb3f904b37d4dc52e3db52f118: Status 404 returned error can't find the container with id 220054fcd716bc0c8fae2c02b34fae40f8fdfbeb3f904b37d4dc52e3db52f118
Sep 11 17:19:29.698759 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:29.698727    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gnrll"]
Sep 11 17:19:29.703021 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:19:29.702977    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaabf0e8_8fd4_4a7d_85b5_1713702add55.slice/crio-f247c3d55ab5750a19f26640393d0b09769bb6dcc28b26c6ca19a97efc4b3569 WatchSource:0}: Error finding container f247c3d55ab5750a19f26640393d0b09769bb6dcc28b26c6ca19a97efc4b3569: Status 404 returned error can't find the container with id f247c3d55ab5750a19f26640393d0b09769bb6dcc28b26c6ca19a97efc4b3569
Sep 11 17:19:30.341580 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:30.341524    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r" event={"ID":"025cbfc6-fdb8-41f9-b574-9c6accc6b576","Type":"ContainerStarted","Data":"59829072a25fc4fb5b1fdba91fce084b0b101011d75ee759b750c241a888844e"}
Sep 11 17:19:30.342041 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:30.341653    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r"
Sep 11 17:19:30.342041 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:30.341704    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r" event={"ID":"025cbfc6-fdb8-41f9-b574-9c6accc6b576","Type":"ContainerStarted","Data":"bc02ba25433da35a57c559c56f9f29d7d5435ce3337adb6ff63d692504ed581e"}
Sep 11 17:19:30.343168 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:30.343142    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gnrll" event={"ID":"daabf0e8-8fd4-4a7d-85b5-1713702add55","Type":"ContainerStarted","Data":"bd5bfda8129fdc1867dac9e478bee4719fd085bc035dba16eb10c7ce28487d57"}
Sep 11 17:19:30.343270 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:30.343188    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gnrll" event={"ID":"daabf0e8-8fd4-4a7d-85b5-1713702add55","Type":"ContainerStarted","Data":"f247c3d55ab5750a19f26640393d0b09769bb6dcc28b26c6ca19a97efc4b3569"}
Sep 11 17:19:30.344203 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:30.344181    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z4bx7" event={"ID":"77f356e7-bfb0-47dc-9910-4d49ef7f2d71","Type":"ContainerStarted","Data":"220054fcd716bc0c8fae2c02b34fae40f8fdfbeb3f904b37d4dc52e3db52f118"}
Sep 11 17:19:30.363485 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:30.363434    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r" podStartSLOduration=62.363415104 podStartE2EDuration="1m2.363415104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:18:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:30.362310167 +0000 UTC m=+50.756227623" watchObservedRunningTime="2026-09-11 17:19:30.363415104 +0000 UTC m=+50.757332561"
Sep 11 17:19:31.348738 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:31.348703    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gnrll" event={"ID":"daabf0e8-8fd4-4a7d-85b5-1713702add55","Type":"ContainerStarted","Data":"0a0ca8dbc9af727ddadfd7576bebfd9e5d76f909c37d1a52166fe78f71316667"}
Sep 11 17:19:32.353755 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:32.353722    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z4bx7" event={"ID":"77f356e7-bfb0-47dc-9910-4d49ef7f2d71","Type":"ContainerStarted","Data":"42b284f81ebc5c67e5ff2138f87101f0f7c4f353614c344c84b6f105d0eea8ea"}
Sep 11 17:19:32.354190 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:32.353777    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z4bx7" event={"ID":"77f356e7-bfb0-47dc-9910-4d49ef7f2d71","Type":"ContainerStarted","Data":"263680c02303d7f76f246401ae35c9d060e8330215027d2b793b7bf0ed9ed562"}
Sep 11 17:19:32.354190 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:32.353862    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-z4bx7"
Sep 11 17:19:32.380207 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:32.380152    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-z4bx7" podStartSLOduration=17.205614352 podStartE2EDuration="19.380136315s" totalImagesPullingTime="2.174521963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:19:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:32.379893605 +0000 UTC m=+52.773811060" watchObservedRunningTime="2026-09-11 17:19:32.380136315 +0000 UTC m=+52.774053769"
Sep 11 17:19:35.365778 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:35.365737    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gnrll" event={"ID":"daabf0e8-8fd4-4a7d-85b5-1713702add55","Type":"ContainerStarted","Data":"ac116369c85d4996ec79b17a20442b2a2a3dc6ebc424eb672487c44af08b41e0"}
Sep 11 17:19:35.387285 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:35.387239    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gnrll" podStartSLOduration=17.94052062 podStartE2EDuration="22.387224841s" totalImagesPullingTime="4.446704221s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:19:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:35.386258723 +0000 UTC m=+55.780176189" watchObservedRunningTime="2026-09-11 17:19:35.387224841 +0000 UTC m=+55.781142326"
Sep 11 17:19:37.213491 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.213457    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-754bd846db-822gr"]
Sep 11 17:19:37.248254 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.248224    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-594f67b5f9-wg7mz"]
Sep 11 17:19:37.248406 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.248349    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-754bd846db-822gr"
Sep 11 17:19:37.251253 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.251227    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 17:19:37.251370 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.251259    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 17:19:37.255794 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.255598    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 11 17:19:37.255794 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.255626    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 17:19:37.255794 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.255697    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-4qx9j\""
Sep 11 17:19:37.262042 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.262023    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-594f67b5f9-wg7mz"]
Sep 11 17:19:37.262126 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.262052    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-754bd846db-822gr"]
Sep 11 17:19:37.262126 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.262113    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594f67b5f9-wg7mz"
Sep 11 17:19:37.263709 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.263688    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5846c5c455-87nvm"]
Sep 11 17:19:37.266847 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.266822    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 11 17:19:37.280803 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.280784    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5846c5c455-87nvm"
Sep 11 17:19:37.283498 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.283479    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 11 17:19:37.283598 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.283569    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 11 17:19:37.283646 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.283626    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 11 17:19:37.289958 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.289938    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 11 17:19:37.300799 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.300779    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5846c5c455-87nvm"]
Sep 11 17:19:37.330609 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.330559    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/50527c8a-26e2-46f1-97a7-157fdafad286-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5846c5c455-87nvm\" (UID: \"50527c8a-26e2-46f1-97a7-157fdafad286\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5846c5c455-87nvm"
Sep 11 17:19:37.330777 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.330632    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w546q\" (UniqueName: \"kubernetes.io/projected/11717ee5-61c4-4930-8993-abca8ebea46f-kube-api-access-w546q\") pod \"klusterlet-addon-workmgr-594f67b5f9-wg7mz\" (UID: \"11717ee5-61c4-4930-8993-abca8ebea46f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594f67b5f9-wg7mz"
Sep 11 17:19:37.330777 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.330663    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/50527c8a-26e2-46f1-97a7-157fdafad286-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5846c5c455-87nvm\" (UID: \"50527c8a-26e2-46f1-97a7-157fdafad286\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5846c5c455-87nvm"
Sep 11 17:19:37.330777 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.330688    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f09fec82-4d68-4edd-bfe4-21d7034c6691-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-754bd846db-822gr\" (UID: \"f09fec82-4d68-4edd-bfe4-21d7034c6691\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-754bd846db-822gr"
Sep 11 17:19:37.330777 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.330720    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/11717ee5-61c4-4930-8993-abca8ebea46f-klusterlet-config\") pod \"klusterlet-addon-workmgr-594f67b5f9-wg7mz\" (UID: \"11717ee5-61c4-4930-8993-abca8ebea46f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594f67b5f9-wg7mz"
Sep 11 17:19:37.330918 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.330787    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/50527c8a-26e2-46f1-97a7-157fdafad286-hub\") pod \"cluster-proxy-proxy-agent-5846c5c455-87nvm\" (UID: \"50527c8a-26e2-46f1-97a7-157fdafad286\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5846c5c455-87nvm"
Sep 11 17:19:37.330918 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.330831    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/50527c8a-26e2-46f1-97a7-157fdafad286-ca\") pod \"cluster-proxy-proxy-agent-5846c5c455-87nvm\" (UID: \"50527c8a-26e2-46f1-97a7-157fdafad286\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5846c5c455-87nvm"
Sep 11 17:19:37.330918 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.330856    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/50527c8a-26e2-46f1-97a7-157fdafad286-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5846c5c455-87nvm\" (UID: \"50527c8a-26e2-46f1-97a7-157fdafad286\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5846c5c455-87nvm"
Sep 11 17:19:37.330918 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.330880    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk657\" (UniqueName: \"kubernetes.io/projected/50527c8a-26e2-46f1-97a7-157fdafad286-kube-api-access-rk657\") pod \"cluster-proxy-proxy-agent-5846c5c455-87nvm\" (UID: \"50527c8a-26e2-46f1-97a7-157fdafad286\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5846c5c455-87nvm"
Sep 11 17:19:37.331097 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.330965    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn2zb\" (UniqueName: \"kubernetes.io/projected/f09fec82-4d68-4edd-bfe4-21d7034c6691-kube-api-access-qn2zb\") pod \"managed-serviceaccount-addon-agent-754bd846db-822gr\" (UID: \"f09fec82-4d68-4edd-bfe4-21d7034c6691\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-754bd846db-822gr"
Sep 11 17:19:37.331097 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.331003    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11717ee5-61c4-4930-8993-abca8ebea46f-tmp\") pod \"klusterlet-addon-workmgr-594f67b5f9-wg7mz\" (UID: \"11717ee5-61c4-4930-8993-abca8ebea46f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594f67b5f9-wg7mz"
Sep 11 17:19:37.331918 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.331899    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d58fbcc7c-65k7r"]
Sep 11 17:19:37.406255 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.406224    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-647bdcc9cf-tsrbr"]
Sep 11 17:19:37.423164 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.423140    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-647bdcc9cf-tsrbr"
Sep 11 17:19:37.431801 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.431778    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/50527c8a-26e2-46f1-97a7-157fdafad286-hub\") pod \"cluster-proxy-proxy-agent-5846c5c455-87nvm\" (UID: \"50527c8a-26e2-46f1-97a7-157fdafad286\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5846c5c455-87nvm"
Sep 11 17:19:37.431909 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.431812    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/76c521a8-16b1-487f-ae42-3f94d0605fcb-registry-certificates\") pod \"image-registry-647bdcc9cf-tsrbr\" (UID: \"76c521a8-16b1-487f-ae42-3f94d0605fcb\") " pod="openshift-image-registry/image-registry-647bdcc9cf-tsrbr"
Sep 11 17:19:37.431909 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.431834    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/50527c8a-26e2-46f1-97a7-157fdafad286-ca\") pod \"cluster-proxy-proxy-agent-5846c5c455-87nvm\" (UID: \"50527c8a-26e2-46f1-97a7-157fdafad286\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5846c5c455-87nvm"
Sep 11 17:19:37.431909 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.431854    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/50527c8a-26e2-46f1-97a7-157fdafad286-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5846c5c455-87nvm\" (UID: \"50527c8a-26e2-46f1-97a7-157fdafad286\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5846c5c455-87nvm"
Sep 11 17:19:37.431909 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.431875    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk657\" (UniqueName: \"kubernetes.io/projected/50527c8a-26e2-46f1-97a7-157fdafad286-kube-api-access-rk657\") pod \"cluster-proxy-proxy-agent-5846c5c455-87nvm\" (UID: \"50527c8a-26e2-46f1-97a7-157fdafad286\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5846c5c455-87nvm"
Sep 11 17:19:37.431909 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.431904    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/76c521a8-16b1-487f-ae42-3f94d0605fcb-bound-sa-token\") pod \"image-registry-647bdcc9cf-tsrbr\" (UID: \"76c521a8-16b1-487f-ae42-3f94d0605fcb\") " pod="openshift-image-registry/image-registry-647bdcc9cf-tsrbr"
Sep 11 17:19:37.432233 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.432033    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn2zb\" (UniqueName: \"kubernetes.io/projected/f09fec82-4d68-4edd-bfe4-21d7034c6691-kube-api-access-qn2zb\") pod \"managed-serviceaccount-addon-agent-754bd846db-822gr\" (UID: \"f09fec82-4d68-4edd-bfe4-21d7034c6691\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-754bd846db-822gr"
Sep 11 17:19:37.432233 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.432067    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11717ee5-61c4-4930-8993-abca8ebea46f-tmp\") pod \"klusterlet-addon-workmgr-594f67b5f9-wg7mz\" (UID: \"11717ee5-61c4-4930-8993-abca8ebea46f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594f67b5f9-wg7mz"
Sep 11 17:19:37.432233 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.432106    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/76c521a8-16b1-487f-ae42-3f94d0605fcb-installation-pull-secrets\") pod \"image-registry-647bdcc9cf-tsrbr\" (UID: \"76c521a8-16b1-487f-ae42-3f94d0605fcb\") " pod="openshift-image-registry/image-registry-647bdcc9cf-tsrbr"
Sep 11 17:19:37.432233 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.432160    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76c521a8-16b1-487f-ae42-3f94d0605fcb-registry-tls\") pod \"image-registry-647bdcc9cf-tsrbr\" (UID: \"76c521a8-16b1-487f-ae42-3f94d0605fcb\") " pod="openshift-image-registry/image-registry-647bdcc9cf-tsrbr"
Sep 11 17:19:37.432233 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.432195    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q6wn\" (UniqueName: \"kubernetes.io/projected/76c521a8-16b1-487f-ae42-3f94d0605fcb-kube-api-access-2q6wn\") pod \"image-registry-647bdcc9cf-tsrbr\" (UID: \"76c521a8-16b1-487f-ae42-3f94d0605fcb\") " pod="openshift-image-registry/image-registry-647bdcc9cf-tsrbr"
Sep 11 17:19:37.432528 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.432283    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76c521a8-16b1-487f-ae42-3f94d0605fcb-trusted-ca\") pod \"image-registry-647bdcc9cf-tsrbr\" (UID: \"76c521a8-16b1-487f-ae42-3f94d0605fcb\") " pod="openshift-image-registry/image-registry-647bdcc9cf-tsrbr"
Sep 11 17:19:37.432528 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.432336    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/50527c8a-26e2-46f1-97a7-157fdafad286-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5846c5c455-87nvm\" (UID: \"50527c8a-26e2-46f1-97a7-157fdafad286\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5846c5c455-87nvm"
Sep 11 17:19:37.432528 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.432364    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/76c521a8-16b1-487f-ae42-3f94d0605fcb-image-registry-private-configuration\") pod \"image-registry-647bdcc9cf-tsrbr\" (UID: \"76c521a8-16b1-487f-ae42-3f94d0605fcb\") " pod="openshift-image-registry/image-registry-647bdcc9cf-tsrbr"
Sep 11 17:19:37.432528 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.432400    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w546q\" (UniqueName: \"kubernetes.io/projected/11717ee5-61c4-4930-8993-abca8ebea46f-kube-api-access-w546q\") pod \"klusterlet-addon-workmgr-594f67b5f9-wg7mz\" (UID: \"11717ee5-61c4-4930-8993-abca8ebea46f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594f67b5f9-wg7mz"
Sep 11 17:19:37.432528 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.432429    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/50527c8a-26e2-46f1-97a7-157fdafad286-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5846c5c455-87nvm\" (UID: \"50527c8a-26e2-46f1-97a7-157fdafad286\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5846c5c455-87nvm"
Sep 11 17:19:37.432528 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.432453    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f09fec82-4d68-4edd-bfe4-21d7034c6691-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-754bd846db-822gr\" (UID: \"f09fec82-4d68-4edd-bfe4-21d7034c6691\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-754bd846db-822gr"
Sep 11 17:19:37.432528 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.432454    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11717ee5-61c4-4930-8993-abca8ebea46f-tmp\") pod \"klusterlet-addon-workmgr-594f67b5f9-wg7mz\" (UID: \"11717ee5-61c4-4930-8993-abca8ebea46f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594f67b5f9-wg7mz"
Sep 11 17:19:37.432528 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.432476    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/76c521a8-16b1-487f-ae42-3f94d0605fcb-ca-trust-extracted\") pod \"image-registry-647bdcc9cf-tsrbr\" (UID: \"76c521a8-16b1-487f-ae42-3f94d0605fcb\") " pod="openshift-image-registry/image-registry-647bdcc9cf-tsrbr"
Sep 11 17:19:37.432528 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.432509    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/11717ee5-61c4-4930-8993-abca8ebea46f-klusterlet-config\") pod \"klusterlet-addon-workmgr-594f67b5f9-wg7mz\" (UID: \"11717ee5-61c4-4930-8993-abca8ebea46f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594f67b5f9-wg7mz"
Sep 11 17:19:37.433205 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.433179    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/50527c8a-26e2-46f1-97a7-157fdafad286-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5846c5c455-87nvm\" (UID: \"50527c8a-26e2-46f1-97a7-157fdafad286\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5846c5c455-87nvm"
Sep 11 17:19:37.434698 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.434671    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/50527c8a-26e2-46f1-97a7-157fdafad286-hub\") pod \"cluster-proxy-proxy-agent-5846c5c455-87nvm\" (UID: \"50527c8a-26e2-46f1-97a7-157fdafad286\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5846c5c455-87nvm"
Sep 11 17:19:37.434802 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.434682    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/50527c8a-26e2-46f1-97a7-157fdafad286-ca\") pod \"cluster-proxy-proxy-agent-5846c5c455-87nvm\" (UID: \"50527c8a-26e2-46f1-97a7-157fdafad286\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5846c5c455-87nvm"
Sep 11 17:19:37.434984 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.434967    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/50527c8a-26e2-46f1-97a7-157fdafad286-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5846c5c455-87nvm\" (UID: \"50527c8a-26e2-46f1-97a7-157fdafad286\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5846c5c455-87nvm"
Sep 11 17:19:37.435173 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.435154    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/50527c8a-26e2-46f1-97a7-157fdafad286-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5846c5c455-87nvm\" (UID: \"50527c8a-26e2-46f1-97a7-157fdafad286\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5846c5c455-87nvm"
Sep 11 17:19:37.435247 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.435182    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/11717ee5-61c4-4930-8993-abca8ebea46f-klusterlet-config\") pod \"klusterlet-addon-workmgr-594f67b5f9-wg7mz\" (UID: \"11717ee5-61c4-4930-8993-abca8ebea46f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594f67b5f9-wg7mz"
Sep 11 17:19:37.435300 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.435258    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f09fec82-4d68-4edd-bfe4-21d7034c6691-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-754bd846db-822gr\" (UID: \"f09fec82-4d68-4edd-bfe4-21d7034c6691\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-754bd846db-822gr"
Sep 11 17:19:37.445671 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.445622    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-647bdcc9cf-tsrbr"]
Sep 11 17:19:37.466763 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.466703    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w546q\" (UniqueName: \"kubernetes.io/projected/11717ee5-61c4-4930-8993-abca8ebea46f-kube-api-access-w546q\") pod \"klusterlet-addon-workmgr-594f67b5f9-wg7mz\" (UID: \"11717ee5-61c4-4930-8993-abca8ebea46f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594f67b5f9-wg7mz"
Sep 11 17:19:37.470450 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.470400    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn2zb\" (UniqueName: \"kubernetes.io/projected/f09fec82-4d68-4edd-bfe4-21d7034c6691-kube-api-access-qn2zb\") pod \"managed-serviceaccount-addon-agent-754bd846db-822gr\" (UID: \"f09fec82-4d68-4edd-bfe4-21d7034c6691\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-754bd846db-822gr"
Sep 11 17:19:37.470634 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.470613    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk657\" (UniqueName: \"kubernetes.io/projected/50527c8a-26e2-46f1-97a7-157fdafad286-kube-api-access-rk657\") pod \"cluster-proxy-proxy-agent-5846c5c455-87nvm\" (UID: \"50527c8a-26e2-46f1-97a7-157fdafad286\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5846c5c455-87nvm"
Sep 11 17:19:37.533286 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.533255    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/76c521a8-16b1-487f-ae42-3f94d0605fcb-image-registry-private-configuration\") pod \"image-registry-647bdcc9cf-tsrbr\" (UID: \"76c521a8-16b1-487f-ae42-3f94d0605fcb\") " pod="openshift-image-registry/image-registry-647bdcc9cf-tsrbr"
Sep 11 17:19:37.533422 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.533294    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/76c521a8-16b1-487f-ae42-3f94d0605fcb-ca-trust-extracted\") pod \"image-registry-647bdcc9cf-tsrbr\" (UID: \"76c521a8-16b1-487f-ae42-3f94d0605fcb\") " pod="openshift-image-registry/image-registry-647bdcc9cf-tsrbr"
Sep 11 17:19:37.533422 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.533332    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/76c521a8-16b1-487f-ae42-3f94d0605fcb-registry-certificates\") pod \"image-registry-647bdcc9cf-tsrbr\" (UID: \"76c521a8-16b1-487f-ae42-3f94d0605fcb\") " pod="openshift-image-registry/image-registry-647bdcc9cf-tsrbr"
Sep 11 17:19:37.533422 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.533363    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/76c521a8-16b1-487f-ae42-3f94d0605fcb-bound-sa-token\") pod \"image-registry-647bdcc9cf-tsrbr\" (UID: \"76c521a8-16b1-487f-ae42-3f94d0605fcb\") " pod="openshift-image-registry/image-registry-647bdcc9cf-tsrbr"
Sep 11 17:19:37.533422 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.533405    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/76c521a8-16b1-487f-ae42-3f94d0605fcb-installation-pull-secrets\") pod \"image-registry-647bdcc9cf-tsrbr\" (UID: \"76c521a8-16b1-487f-ae42-3f94d0605fcb\") " pod="openshift-image-registry/image-registry-647bdcc9cf-tsrbr"
Sep 11 17:19:37.533612 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.533444    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76c521a8-16b1-487f-ae42-3f94d0605fcb-registry-tls\") pod \"image-registry-647bdcc9cf-tsrbr\" (UID: \"76c521a8-16b1-487f-ae42-3f94d0605fcb\") " pod="openshift-image-registry/image-registry-647bdcc9cf-tsrbr"
Sep 11 17:19:37.533612 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.533472    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q6wn\" (UniqueName: \"kubernetes.io/projected/76c521a8-16b1-487f-ae42-3f94d0605fcb-kube-api-access-2q6wn\") pod \"image-registry-647bdcc9cf-tsrbr\" (UID: \"76c521a8-16b1-487f-ae42-3f94d0605fcb\") " pod="openshift-image-registry/image-registry-647bdcc9cf-tsrbr"
Sep 11 17:19:37.533612 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.533542    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76c521a8-16b1-487f-ae42-3f94d0605fcb-trusted-ca\") pod \"image-registry-647bdcc9cf-tsrbr\" (UID: \"76c521a8-16b1-487f-ae42-3f94d0605fcb\") " pod="openshift-image-registry/image-registry-647bdcc9cf-tsrbr"
Sep 11 17:19:37.533776 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.533687    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/76c521a8-16b1-487f-ae42-3f94d0605fcb-ca-trust-extracted\") pod \"image-registry-647bdcc9cf-tsrbr\" (UID: \"76c521a8-16b1-487f-ae42-3f94d0605fcb\") " pod="openshift-image-registry/image-registry-647bdcc9cf-tsrbr"
Sep 11 17:19:37.534131 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.534107    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/76c521a8-16b1-487f-ae42-3f94d0605fcb-registry-certificates\") pod \"image-registry-647bdcc9cf-tsrbr\" (UID: \"76c521a8-16b1-487f-ae42-3f94d0605fcb\") " pod="openshift-image-registry/image-registry-647bdcc9cf-tsrbr"
Sep 11 17:19:37.534495 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.534476    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76c521a8-16b1-487f-ae42-3f94d0605fcb-trusted-ca\") pod \"image-registry-647bdcc9cf-tsrbr\" (UID: \"76c521a8-16b1-487f-ae42-3f94d0605fcb\") " pod="openshift-image-registry/image-registry-647bdcc9cf-tsrbr"
Sep 11 17:19:37.535787 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.535765    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/76c521a8-16b1-487f-ae42-3f94d0605fcb-installation-pull-secrets\") pod \"image-registry-647bdcc9cf-tsrbr\" (UID: \"76c521a8-16b1-487f-ae42-3f94d0605fcb\") " pod="openshift-image-registry/image-registry-647bdcc9cf-tsrbr"
Sep 11 17:19:37.536095 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.536077    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/76c521a8-16b1-487f-ae42-3f94d0605fcb-image-registry-private-configuration\") pod \"image-registry-647bdcc9cf-tsrbr\" (UID: \"76c521a8-16b1-487f-ae42-3f94d0605fcb\") " pod="openshift-image-registry/image-registry-647bdcc9cf-tsrbr"
Sep 11 17:19:37.536202 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.536185    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76c521a8-16b1-487f-ae42-3f94d0605fcb-registry-tls\") pod \"image-registry-647bdcc9cf-tsrbr\" (UID: \"76c521a8-16b1-487f-ae42-3f94d0605fcb\") " pod="openshift-image-registry/image-registry-647bdcc9cf-tsrbr"
Sep 11 17:19:37.548160 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.548141    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/76c521a8-16b1-487f-ae42-3f94d0605fcb-bound-sa-token\") pod \"image-registry-647bdcc9cf-tsrbr\" (UID: \"76c521a8-16b1-487f-ae42-3f94d0605fcb\") " pod="openshift-image-registry/image-registry-647bdcc9cf-tsrbr"
Sep 11 17:19:37.548915 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.548893    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q6wn\" (UniqueName: \"kubernetes.io/projected/76c521a8-16b1-487f-ae42-3f94d0605fcb-kube-api-access-2q6wn\") pod \"image-registry-647bdcc9cf-tsrbr\" (UID: \"76c521a8-16b1-487f-ae42-3f94d0605fcb\") " pod="openshift-image-registry/image-registry-647bdcc9cf-tsrbr"
Sep 11 17:19:37.569736 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.569709    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-754bd846db-822gr"
Sep 11 17:19:37.576835 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.576818    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594f67b5f9-wg7mz"
Sep 11 17:19:37.588682 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.588659    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5846c5c455-87nvm"
Sep 11 17:19:37.779797 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.779718    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-647bdcc9cf-tsrbr"
Sep 11 17:19:37.900909 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:19:37.900882    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf09fec82_4d68_4edd_bfe4_21d7034c6691.slice/crio-b9c245176d022bbd3c5d26c1262deed0b84f3f84e06331ce66ba37595c78e74b WatchSource:0}: Error finding container b9c245176d022bbd3c5d26c1262deed0b84f3f84e06331ce66ba37595c78e74b: Status 404 returned error can't find the container with id b9c245176d022bbd3c5d26c1262deed0b84f3f84e06331ce66ba37595c78e74b
Sep 11 17:19:37.902416 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.902395    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-754bd846db-822gr"]
Sep 11 17:19:37.942300 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.942274    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-594f67b5f9-wg7mz"]
Sep 11 17:19:37.942524 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:19:37.942499    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50527c8a_26e2_46f1_97a7_157fdafad286.slice/crio-0ad26fcde5b788cfeba2832d3f55f6ed1eca2d9e306879534942cc15c7eb0e42 WatchSource:0}: Error finding container 0ad26fcde5b788cfeba2832d3f55f6ed1eca2d9e306879534942cc15c7eb0e42: Status 404 returned error can't find the container with id 0ad26fcde5b788cfeba2832d3f55f6ed1eca2d9e306879534942cc15c7eb0e42
Sep 11 17:19:37.943284 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:37.943183    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5846c5c455-87nvm"]
Sep 11 17:19:37.943284 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:19:37.943197    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11717ee5_61c4_4930_8993_abca8ebea46f.slice/crio-253645cfbb5bc6565a44ec07274bc2736cce38301c0668be49e037f679c00086 WatchSource:0}: Error finding container 253645cfbb5bc6565a44ec07274bc2736cce38301c0668be49e037f679c00086: Status 404 returned error can't find the container with id 253645cfbb5bc6565a44ec07274bc2736cce38301c0668be49e037f679c00086
Sep 11 17:19:38.122057 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:19:38.122024    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76c521a8_16b1_487f_ae42_3f94d0605fcb.slice/crio-4829cb720197fb724ee3adb5dc17add8e1ad4959a54061df36d168a18ab00963 WatchSource:0}: Error finding container 4829cb720197fb724ee3adb5dc17add8e1ad4959a54061df36d168a18ab00963: Status 404 returned error can't find the container with id 4829cb720197fb724ee3adb5dc17add8e1ad4959a54061df36d168a18ab00963
Sep 11 17:19:38.123881 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:38.123863    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-647bdcc9cf-tsrbr"]
Sep 11 17:19:38.376109 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:38.376027    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594f67b5f9-wg7mz" event={"ID":"11717ee5-61c4-4930-8993-abca8ebea46f","Type":"ContainerStarted","Data":"253645cfbb5bc6565a44ec07274bc2736cce38301c0668be49e037f679c00086"}
Sep 11 17:19:38.377370 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:38.377337    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5846c5c455-87nvm" event={"ID":"50527c8a-26e2-46f1-97a7-157fdafad286","Type":"ContainerStarted","Data":"0ad26fcde5b788cfeba2832d3f55f6ed1eca2d9e306879534942cc15c7eb0e42"}
Sep 11 17:19:38.378945 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:38.378910    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-647bdcc9cf-tsrbr" event={"ID":"76c521a8-16b1-487f-ae42-3f94d0605fcb","Type":"ContainerStarted","Data":"77ca30356d0f159bc0f067b978aad35f943e5cd8a0a7069f1f7375907e7adab3"}
Sep 11 17:19:38.379082 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:38.378964    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-647bdcc9cf-tsrbr" event={"ID":"76c521a8-16b1-487f-ae42-3f94d0605fcb","Type":"ContainerStarted","Data":"4829cb720197fb724ee3adb5dc17add8e1ad4959a54061df36d168a18ab00963"}
Sep 11 17:19:38.379082 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:38.379019    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-647bdcc9cf-tsrbr"
Sep 11 17:19:38.380152 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:38.380128    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-754bd846db-822gr" event={"ID":"f09fec82-4d68-4edd-bfe4-21d7034c6691","Type":"ContainerStarted","Data":"b9c245176d022bbd3c5d26c1262deed0b84f3f84e06331ce66ba37595c78e74b"}
Sep 11 17:19:38.421362 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:38.421277    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-647bdcc9cf-tsrbr" podStartSLOduration=1.421254349 podStartE2EDuration="1.421254349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:19:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:38.420342875 +0000 UTC m=+58.814260334" watchObservedRunningTime="2026-09-11 17:19:38.421254349 +0000 UTC m=+58.815171805"
Sep 11 17:19:39.282282 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:39.282247    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wr5ps"
Sep 11 17:19:41.393073 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:41.393031    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-754bd846db-822gr" event={"ID":"f09fec82-4d68-4edd-bfe4-21d7034c6691","Type":"ContainerStarted","Data":"8f7158160786f7b2e55a7580d4f4a8f1f6cb6fdf5801768e46d0013334e87d91"}
Sep 11 17:19:41.431038 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:41.430969    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-754bd846db-822gr" podStartSLOduration=1.8069851959999998 podStartE2EDuration="4.430951705s" totalImagesPullingTime="2.623966509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:19:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:41.430286945 +0000 UTC m=+61.824204401" watchObservedRunningTime="2026-09-11 17:19:41.430951705 +0000 UTC m=+61.824869160"
Sep 11 17:19:42.361387 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:42.360519    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-z4bx7"
Sep 11 17:19:45.810702 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:45.810615    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85715083-7220-4a0d-9dd0-426a131d8440-original-pull-secret\") pod \"global-pull-secret-syncer-4lmxn\" (UID: \"85715083-7220-4a0d-9dd0-426a131d8440\") " pod="kube-system/global-pull-secret-syncer-4lmxn"
Sep 11 17:19:45.810702 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:45.810677    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a382e1b1-e35e-4975-823b-0e9efe9a84f3-metrics-certs\") pod \"network-metrics-daemon-scdq8\" (UID: \"a382e1b1-e35e-4975-823b-0e9efe9a84f3\") " pod="openshift-multus/network-metrics-daemon-scdq8"
Sep 11 17:19:45.817593 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:45.817573    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 17:19:45.818140 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:45.818123    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 17:19:45.823864 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:45.823840    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85715083-7220-4a0d-9dd0-426a131d8440-original-pull-secret\") pod \"global-pull-secret-syncer-4lmxn\" (UID: \"85715083-7220-4a0d-9dd0-426a131d8440\") " pod="kube-system/global-pull-secret-syncer-4lmxn"
Sep 11 17:19:45.824010 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:45.823850    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a382e1b1-e35e-4975-823b-0e9efe9a84f3-metrics-certs\") pod \"network-metrics-daemon-scdq8\" (UID: \"a382e1b1-e35e-4975-823b-0e9efe9a84f3\") " pod="openshift-multus/network-metrics-daemon-scdq8"
Sep 11 17:19:45.944676 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:45.944644    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gdccs\""
Sep 11 17:19:45.947265 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:45.947244    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4lmxn"
Sep 11 17:19:45.951294 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:45.951279    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-scdq8"
Sep 11 17:19:46.013411 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:46.012924    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjvvx\" (UniqueName: \"kubernetes.io/projected/fab8e5a6-345d-4e9d-984e-3019fb93b9a9-kube-api-access-mjvvx\") pod \"network-check-target-frlv9\" (UID: \"fab8e5a6-345d-4e9d-984e-3019fb93b9a9\") " pod="openshift-network-diagnostics/network-check-target-frlv9"
Sep 11 17:19:46.018594 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:46.018276    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 17:19:46.026447 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:46.026423    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 17:19:46.037942 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:46.037884    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjvvx\" (UniqueName: \"kubernetes.io/projected/fab8e5a6-345d-4e9d-984e-3019fb93b9a9-kube-api-access-mjvvx\") pod \"network-check-target-frlv9\" (UID: \"fab8e5a6-345d-4e9d-984e-3019fb93b9a9\") " pod="openshift-network-diagnostics/network-check-target-frlv9"
Sep 11 17:19:46.223494 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:46.223461    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4lmxn"]
Sep 11 17:19:46.226796 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:19:46.226762    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85715083_7220_4a0d_9dd0_426a131d8440.slice/crio-4cf9af17170e587be4b11c2eb8ec840e512eb5e5945a56ad57288d9d6102a875 WatchSource:0}: Error finding container 4cf9af17170e587be4b11c2eb8ec840e512eb5e5945a56ad57288d9d6102a875: Status 404 returned error can't find the container with id 4cf9af17170e587be4b11c2eb8ec840e512eb5e5945a56ad57288d9d6102a875
Sep 11 17:19:46.227455 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:46.227431    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-scdq8"]
Sep 11 17:19:46.230125 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:19:46.230102    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda382e1b1_e35e_4975_823b_0e9efe9a84f3.slice/crio-b40db9b0af82367383cdd04dc59911fb6e690739464c1836077bd36adae051f8 WatchSource:0}: Error finding container b40db9b0af82367383cdd04dc59911fb6e690739464c1836077bd36adae051f8: Status 404 returned error can't find the container with id b40db9b0af82367383cdd04dc59911fb6e690739464c1836077bd36adae051f8
Sep 11 17:19:46.236508 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:46.236491    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vthbj\""
Sep 11 17:19:46.244320 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:46.244303    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-frlv9"
Sep 11 17:19:46.412894 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:46.412854    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594f67b5f9-wg7mz" event={"ID":"11717ee5-61c4-4930-8993-abca8ebea46f","Type":"ContainerStarted","Data":"8b65b3b4c90ab633848f2fb7377986973baabdbd04d313e3c60bfb2134f0f1f3"}
Sep 11 17:19:46.413077 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:46.412915    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594f67b5f9-wg7mz"
Sep 11 17:19:46.415137 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:46.415117    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594f67b5f9-wg7mz"
Sep 11 17:19:46.415279 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:46.415259    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5846c5c455-87nvm" event={"ID":"50527c8a-26e2-46f1-97a7-157fdafad286","Type":"ContainerStarted","Data":"850226ec3d21f859bc68874d7b749fc28583f4182a2fc7595234c3a05ae959c2"}
Sep 11 17:19:46.415320 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:46.415295    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5846c5c455-87nvm" event={"ID":"50527c8a-26e2-46f1-97a7-157fdafad286","Type":"ContainerStarted","Data":"12748297b6278746cbac2d28d9f3852f9e793cce588a92cc0afbce5c83620372"}
Sep 11 17:19:46.415320 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:46.415314    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5846c5c455-87nvm" event={"ID":"50527c8a-26e2-46f1-97a7-157fdafad286","Type":"ContainerStarted","Data":"5fb85b0c426d3eaa89a2744ae02e997051446550889a3ec70334a12e80c72ba2"}
Sep 11 17:19:46.416429 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:46.416410    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-scdq8" event={"ID":"a382e1b1-e35e-4975-823b-0e9efe9a84f3","Type":"ContainerStarted","Data":"b40db9b0af82367383cdd04dc59911fb6e690739464c1836077bd36adae051f8"}
Sep 11 17:19:46.417535 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:46.417514    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4lmxn" event={"ID":"85715083-7220-4a0d-9dd0-426a131d8440","Type":"ContainerStarted","Data":"4cf9af17170e587be4b11c2eb8ec840e512eb5e5945a56ad57288d9d6102a875"}
Sep 11 17:19:46.434191 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:46.434147    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594f67b5f9-wg7mz" podStartSLOduration=1.930802757 podStartE2EDuration="9.43413399s" totalImagesPullingTime="7.503331233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:19:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:46.433848883 +0000 UTC m=+66.827766341" watchObservedRunningTime="2026-09-11 17:19:46.43413399 +0000 UTC m=+66.828051445"
Sep 11 17:19:46.519856 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:46.519721    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5846c5c455-87nvm" podStartSLOduration=2.024637871 podStartE2EDuration="9.519702658s" totalImagesPullingTime="7.495064787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:19:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:46.518151434 +0000 UTC m=+66.912068886" watchObservedRunningTime="2026-09-11 17:19:46.519702658 +0000 UTC m=+66.913620113"
Sep 11 17:19:46.536310 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:46.536282    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-frlv9"]
Sep 11 17:19:46.539597 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:19:46.539570    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfab8e5a6_345d_4e9d_984e_3019fb93b9a9.slice/crio-43d8c616b7d1e4ec78f935c9b5d3324555442675f0f05dd8caef4d2dc25b3294 WatchSource:0}: Error finding container 43d8c616b7d1e4ec78f935c9b5d3324555442675f0f05dd8caef4d2dc25b3294: Status 404 returned error can't find the container with id 43d8c616b7d1e4ec78f935c9b5d3324555442675f0f05dd8caef4d2dc25b3294
Sep 11 17:19:47.339445 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:47.339387    2562 patch_prober.go:28] interesting pod/image-registry-5d58fbcc7c-65k7r container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 11 17:19:47.339865 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:47.339460    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r" podUID="025cbfc6-fdb8-41f9-b574-9c6accc6b576" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 17:19:47.422605 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:47.422567    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-frlv9" event={"ID":"fab8e5a6-345d-4e9d-984e-3019fb93b9a9","Type":"ContainerStarted","Data":"43d8c616b7d1e4ec78f935c9b5d3324555442675f0f05dd8caef4d2dc25b3294"}
Sep 11 17:19:48.428310 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:48.428269    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-scdq8" event={"ID":"a382e1b1-e35e-4975-823b-0e9efe9a84f3","Type":"ContainerStarted","Data":"bfbc9c56aeb592d6b96069dfc6e04584ef8b10ade12a8ac4167d92b990873f57"}
Sep 11 17:19:48.428723 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:48.428328    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-scdq8" event={"ID":"a382e1b1-e35e-4975-823b-0e9efe9a84f3","Type":"ContainerStarted","Data":"08e8cf30263489f7b055a1f0d1f702ed2c906e8ffbf2bb0564c2cc2495c27215"}
Sep 11 17:19:48.448918 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:48.448860    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-scdq8" podStartSLOduration=67.264881541 podStartE2EDuration="1m8.44884022s" totalImagesPullingTime="1.183958679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:18:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:48.447957585 +0000 UTC m=+68.841875037" watchObservedRunningTime="2026-09-11 17:19:48.44884022 +0000 UTC m=+68.842757676"
Sep 11 17:19:49.640783 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:49.639326    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6xfx5"]
Sep 11 17:19:49.662671 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:49.662606    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6xfx5"
Sep 11 17:19:49.665811 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:49.665368    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 17:19:49.665811 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:49.665623    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 17:19:49.666524 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:49.666048    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rqg8q\""
Sep 11 17:19:49.666524 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:49.666232    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 17:19:49.666524 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:49.666262    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 17:19:49.666524 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:49.666365    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 17:19:49.666766 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:49.666734    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 17:19:49.743966 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:49.743925    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbwvc\" (UniqueName: \"kubernetes.io/projected/d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d-kube-api-access-qbwvc\") pod \"node-exporter-6xfx5\" (UID: \"d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d\") " pod="openshift-monitoring/node-exporter-6xfx5"
Sep 11 17:19:49.744170 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:49.743976    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d-node-exporter-tls\") pod \"node-exporter-6xfx5\" (UID: \"d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d\") " pod="openshift-monitoring/node-exporter-6xfx5"
Sep 11 17:19:49.744170 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:49.744026    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d-root\") pod \"node-exporter-6xfx5\" (UID: \"d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d\") " pod="openshift-monitoring/node-exporter-6xfx5"
Sep 11 17:19:49.744170 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:49.744069    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d-metrics-client-ca\") pod \"node-exporter-6xfx5\" (UID: \"d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d\") " pod="openshift-monitoring/node-exporter-6xfx5"
Sep 11 17:19:49.744170 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:49.744100    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d-node-exporter-textfile\") pod \"node-exporter-6xfx5\" (UID: \"d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d\") " pod="openshift-monitoring/node-exporter-6xfx5"
Sep 11 17:19:49.744170 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:49.744127    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d-node-exporter-wtmp\") pod \"node-exporter-6xfx5\" (UID: \"d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d\") " pod="openshift-monitoring/node-exporter-6xfx5"
Sep 11 17:19:49.744437 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:49.744223    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d-node-exporter-accelerators-collector-config\") pod \"node-exporter-6xfx5\" (UID: \"d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d\") " pod="openshift-monitoring/node-exporter-6xfx5"
Sep 11 17:19:49.744437 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:49.744250    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6xfx5\" (UID: \"d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d\") " pod="openshift-monitoring/node-exporter-6xfx5"
Sep 11 17:19:49.744437 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:49.744299    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d-sys\") pod \"node-exporter-6xfx5\" (UID: \"d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d\") " pod="openshift-monitoring/node-exporter-6xfx5"
Sep 11 17:19:49.845018 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:49.844869    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d-sys\") pod \"node-exporter-6xfx5\" (UID: \"d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d\") " pod="openshift-monitoring/node-exporter-6xfx5"
Sep 11 17:19:49.845018 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:49.844937    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbwvc\" (UniqueName: \"kubernetes.io/projected/d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d-kube-api-access-qbwvc\") pod \"node-exporter-6xfx5\" (UID: \"d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d\") " pod="openshift-monitoring/node-exporter-6xfx5"
Sep 11 17:19:49.845018 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:49.844969    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d-node-exporter-tls\") pod \"node-exporter-6xfx5\" (UID: \"d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d\") " pod="openshift-monitoring/node-exporter-6xfx5"
Sep 11 17:19:49.845018 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:49.845023    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d-root\") pod \"node-exporter-6xfx5\" (UID: \"d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d\") " pod="openshift-monitoring/node-exporter-6xfx5"
Sep 11 17:19:49.845323 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:49.845055    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d-metrics-client-ca\") pod \"node-exporter-6xfx5\" (UID: \"d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d\") " pod="openshift-monitoring/node-exporter-6xfx5"
Sep 11 17:19:49.845323 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:49.845088    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d-node-exporter-textfile\") pod \"node-exporter-6xfx5\" (UID: \"d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d\") " pod="openshift-monitoring/node-exporter-6xfx5"
Sep 11 17:19:49.845323 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:49.845115    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d-node-exporter-wtmp\") pod \"node-exporter-6xfx5\" (UID: \"d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d\") " pod="openshift-monitoring/node-exporter-6xfx5"
Sep 11 17:19:49.845323 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:49.845157    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d-node-exporter-accelerators-collector-config\") pod \"node-exporter-6xfx5\" (UID: \"d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d\") " pod="openshift-monitoring/node-exporter-6xfx5"
Sep 11 17:19:49.845323 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:49.845189    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6xfx5\" (UID: \"d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d\") " pod="openshift-monitoring/node-exporter-6xfx5"
Sep 11 17:19:49.846015 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:49.845712    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d-root\") pod \"node-exporter-6xfx5\" (UID: \"d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d\") " pod="openshift-monitoring/node-exporter-6xfx5"
Sep 11 17:19:49.846015 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:49.845797    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d-sys\") pod \"node-exporter-6xfx5\" (UID: \"d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d\") " pod="openshift-monitoring/node-exporter-6xfx5"
Sep 11 17:19:49.846196 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:49.846185    2562 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 11 17:19:49.847014 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:19:49.846255    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d-node-exporter-tls podName:d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d nodeName:}" failed. No retries permitted until 2026-09-11 17:19:50.346235048 +0000 UTC m=+70.740152495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d-node-exporter-tls") pod "node-exporter-6xfx5" (UID: "d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d") : secret "node-exporter-tls" not found
Sep 11 17:19:49.847014 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:49.846741    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d-node-exporter-textfile\") pod \"node-exporter-6xfx5\" (UID: \"d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d\") " pod="openshift-monitoring/node-exporter-6xfx5"
Sep 11 17:19:49.850018 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:49.847325    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d-metrics-client-ca\") pod \"node-exporter-6xfx5\" (UID: \"d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d\") " pod="openshift-monitoring/node-exporter-6xfx5"
Sep 11 17:19:49.850018 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:49.847470    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d-node-exporter-wtmp\") pod \"node-exporter-6xfx5\" (UID: \"d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d\") " pod="openshift-monitoring/node-exporter-6xfx5"
Sep 11 17:19:49.850018 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:49.847926    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d-node-exporter-accelerators-collector-config\") pod \"node-exporter-6xfx5\" (UID: \"d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d\") " pod="openshift-monitoring/node-exporter-6xfx5"
Sep 11 17:19:49.862809 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:49.862744    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6xfx5\" (UID: \"d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d\") " pod="openshift-monitoring/node-exporter-6xfx5"
Sep 11 17:19:49.883470 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:49.872977    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbwvc\" (UniqueName: \"kubernetes.io/projected/d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d-kube-api-access-qbwvc\") pod \"node-exporter-6xfx5\" (UID: \"d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d\") " pod="openshift-monitoring/node-exporter-6xfx5"
Sep 11 17:19:50.348390 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:50.348308    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d-node-exporter-tls\") pod \"node-exporter-6xfx5\" (UID: \"d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d\") " pod="openshift-monitoring/node-exporter-6xfx5"
Sep 11 17:19:50.351042 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:50.351016    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d-node-exporter-tls\") pod \"node-exporter-6xfx5\" (UID: \"d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d\") " pod="openshift-monitoring/node-exporter-6xfx5"
Sep 11 17:19:50.575360 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:50.575326    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6xfx5"
Sep 11 17:19:52.235189 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:19:52.235155    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3a189bc_8f76_4ad7_95aa_5d006ca0dc3d.slice/crio-f774fb37e9cdda67ac44c3a384de510853a54c443a19af3ccd2693c6efbd2159 WatchSource:0}: Error finding container f774fb37e9cdda67ac44c3a384de510853a54c443a19af3ccd2693c6efbd2159: Status 404 returned error can't find the container with id f774fb37e9cdda67ac44c3a384de510853a54c443a19af3ccd2693c6efbd2159
Sep 11 17:19:52.444325 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:52.444234    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4lmxn" event={"ID":"85715083-7220-4a0d-9dd0-426a131d8440","Type":"ContainerStarted","Data":"0791ee2f55cb1c5b746863a398fa67fb897175e08dd299a364c9982f0525ab0d"}
Sep 11 17:19:52.445642 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:52.445614    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6xfx5" event={"ID":"d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d","Type":"ContainerStarted","Data":"f774fb37e9cdda67ac44c3a384de510853a54c443a19af3ccd2693c6efbd2159"}
Sep 11 17:19:52.446960 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:52.446924    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-frlv9" event={"ID":"fab8e5a6-345d-4e9d-984e-3019fb93b9a9","Type":"ContainerStarted","Data":"4ce5be218b7c76367a9d9e6b17ca864c8caa05691d7af4cb748de46eac779f0b"}
Sep 11 17:19:52.447107 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:52.447036    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-frlv9"
Sep 11 17:19:52.484611 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:52.484566    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4lmxn" podStartSLOduration=66.481320616 podStartE2EDuration="1m12.484549499s" totalImagesPullingTime="6.003228883s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:18:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:52.483060471 +0000 UTC m=+72.876977926" watchObservedRunningTime="2026-09-11 17:19:52.484549499 +0000 UTC m=+72.878466954"
Sep 11 17:19:52.512777 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:52.512728    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-frlv9" podStartSLOduration=66.810813234 podStartE2EDuration="1m12.512712058s" totalImagesPullingTime="5.701898824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:18:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:52.512078504 +0000 UTC m=+72.905995960" watchObservedRunningTime="2026-09-11 17:19:52.512712058 +0000 UTC m=+72.906629513"
Sep 11 17:19:54.455429 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:54.455397    2562 generic.go:356] "Generic (PLEG): container finished" podID="d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d" containerID="f700a44e555d6d1ea646a89844092612aab80a3ef200cb7151b14126c87813b7" exitCode=0
Sep 11 17:19:54.455831 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:54.455492    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6xfx5" event={"ID":"d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d","Type":"ContainerDied","Data":"f700a44e555d6d1ea646a89844092612aab80a3ef200cb7151b14126c87813b7"}
Sep 11 17:19:55.461041 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:55.461006    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6xfx5" event={"ID":"d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d","Type":"ContainerStarted","Data":"30c1817fffe1133550da7208ec7f75cbfd2af4a7c4797d3e79239480bee3f8f8"}
Sep 11 17:19:55.461416 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:55.461057    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6xfx5" event={"ID":"d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d","Type":"ContainerStarted","Data":"1416f0acc3d7d61a830775d71ed7a9d122574ef34a0ffdd877960c759c16f02a"}
Sep 11 17:19:55.490009 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:55.489950    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6xfx5" podStartSLOduration=5.271203962 podStartE2EDuration="6.489934559s" totalImagesPullingTime="1.157155243s" totalInitContainerRuntime="61.575354ms" isStatefulPod=true podCreationTimestamp="2026-09-11 17:19:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:55.488322239 +0000 UTC m=+75.882239695" watchObservedRunningTime="2026-09-11 17:19:55.489934559 +0000 UTC m=+75.883852013"
Sep 11 17:19:57.336215 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:57.336186    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r"
Sep 11 17:19:59.388540 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:19:59.388509    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-647bdcc9cf-tsrbr"
Sep 11 17:20:02.351081 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:20:02.350926    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r" podUID="025cbfc6-fdb8-41f9-b574-9c6accc6b576" containerName="registry" containerID="cri-o://59829072a25fc4fb5b1fdba91fce084b0b101011d75ee759b750c241a888844e" gracePeriod=30
Sep 11 17:20:02.489246 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:20:02.489215    2562 generic.go:356] "Generic (PLEG): container finished" podID="025cbfc6-fdb8-41f9-b574-9c6accc6b576" containerID="59829072a25fc4fb5b1fdba91fce084b0b101011d75ee759b750c241a888844e" exitCode=0
Sep 11 17:20:02.489376 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:20:02.489289    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r" event={"ID":"025cbfc6-fdb8-41f9-b574-9c6accc6b576","Type":"ContainerDied","Data":"59829072a25fc4fb5b1fdba91fce084b0b101011d75ee759b750c241a888844e"}
Sep 11 17:20:02.586590 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:20:02.586568    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r"
Sep 11 17:20:02.745972 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:20:02.745904    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/025cbfc6-fdb8-41f9-b574-9c6accc6b576-installation-pull-secrets\") pod \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\" (UID: \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\") "
Sep 11 17:20:02.745972 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:20:02.745944    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/025cbfc6-fdb8-41f9-b574-9c6accc6b576-registry-tls\") pod \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\" (UID: \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\") "
Sep 11 17:20:02.745972 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:20:02.745961    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/025cbfc6-fdb8-41f9-b574-9c6accc6b576-bound-sa-token\") pod \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\" (UID: \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\") "
Sep 11 17:20:02.746263 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:20:02.746023    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vr4lh\" (UniqueName: \"kubernetes.io/projected/025cbfc6-fdb8-41f9-b574-9c6accc6b576-kube-api-access-vr4lh\") pod \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\" (UID: \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\") "
Sep 11 17:20:02.746263 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:20:02.746050    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/025cbfc6-fdb8-41f9-b574-9c6accc6b576-image-registry-private-configuration\") pod \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\" (UID: \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\") "
Sep 11 17:20:02.746263 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:20:02.746084    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/025cbfc6-fdb8-41f9-b574-9c6accc6b576-ca-trust-extracted\") pod \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\" (UID: \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\") "
Sep 11 17:20:02.746263 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:20:02.746109    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/025cbfc6-fdb8-41f9-b574-9c6accc6b576-registry-certificates\") pod \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\" (UID: \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\") "
Sep 11 17:20:02.746263 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:20:02.746129    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/025cbfc6-fdb8-41f9-b574-9c6accc6b576-trusted-ca\") pod \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\" (UID: \"025cbfc6-fdb8-41f9-b574-9c6accc6b576\") "
Sep 11 17:20:02.746799 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:20:02.746736    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/025cbfc6-fdb8-41f9-b574-9c6accc6b576-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "025cbfc6-fdb8-41f9-b574-9c6accc6b576" (UID: "025cbfc6-fdb8-41f9-b574-9c6accc6b576"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:20:02.746988 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:20:02.746955    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/025cbfc6-fdb8-41f9-b574-9c6accc6b576-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "025cbfc6-fdb8-41f9-b574-9c6accc6b576" (UID: "025cbfc6-fdb8-41f9-b574-9c6accc6b576"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:20:02.748957 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:20:02.748891    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/025cbfc6-fdb8-41f9-b574-9c6accc6b576-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "025cbfc6-fdb8-41f9-b574-9c6accc6b576" (UID: "025cbfc6-fdb8-41f9-b574-9c6accc6b576"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:20:02.749126 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:20:02.749094    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/025cbfc6-fdb8-41f9-b574-9c6accc6b576-kube-api-access-vr4lh" (OuterVolumeSpecName: "kube-api-access-vr4lh") pod "025cbfc6-fdb8-41f9-b574-9c6accc6b576" (UID: "025cbfc6-fdb8-41f9-b574-9c6accc6b576"). InnerVolumeSpecName "kube-api-access-vr4lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:20:02.749249 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:20:02.749226    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/025cbfc6-fdb8-41f9-b574-9c6accc6b576-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "025cbfc6-fdb8-41f9-b574-9c6accc6b576" (UID: "025cbfc6-fdb8-41f9-b574-9c6accc6b576"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:20:02.749316 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:20:02.749266    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/025cbfc6-fdb8-41f9-b574-9c6accc6b576-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "025cbfc6-fdb8-41f9-b574-9c6accc6b576" (UID: "025cbfc6-fdb8-41f9-b574-9c6accc6b576"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:20:02.749368 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:20:02.749336    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/025cbfc6-fdb8-41f9-b574-9c6accc6b576-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "025cbfc6-fdb8-41f9-b574-9c6accc6b576" (UID: "025cbfc6-fdb8-41f9-b574-9c6accc6b576"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:20:02.757978 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:20:02.757947    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025cbfc6-fdb8-41f9-b574-9c6accc6b576-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "025cbfc6-fdb8-41f9-b574-9c6accc6b576" (UID: "025cbfc6-fdb8-41f9-b574-9c6accc6b576"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:20:02.847066 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:20:02.847036    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/025cbfc6-fdb8-41f9-b574-9c6accc6b576-registry-tls\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:20:02.847066 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:20:02.847064    2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/025cbfc6-fdb8-41f9-b574-9c6accc6b576-bound-sa-token\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:20:02.847243 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:20:02.847073    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr4lh\" (UniqueName: \"kubernetes.io/projected/025cbfc6-fdb8-41f9-b574-9c6accc6b576-kube-api-access-vr4lh\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:20:02.847243 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:20:02.847085    2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/025cbfc6-fdb8-41f9-b574-9c6accc6b576-image-registry-private-configuration\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:20:02.847243 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:20:02.847094    2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/025cbfc6-fdb8-41f9-b574-9c6accc6b576-ca-trust-extracted\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:20:02.847243 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:20:02.847103    2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/025cbfc6-fdb8-41f9-b574-9c6accc6b576-registry-certificates\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:20:02.847243 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:20:02.847111    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/025cbfc6-fdb8-41f9-b574-9c6accc6b576-trusted-ca\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:20:02.847243 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:20:02.847119    2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/025cbfc6-fdb8-41f9-b574-9c6accc6b576-installation-pull-secrets\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:20:03.494691 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:20:03.494663    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r"
Sep 11 17:20:03.495131 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:20:03.494662    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d58fbcc7c-65k7r" event={"ID":"025cbfc6-fdb8-41f9-b574-9c6accc6b576","Type":"ContainerDied","Data":"bc02ba25433da35a57c559c56f9f29d7d5435ce3337adb6ff63d692504ed581e"}
Sep 11 17:20:03.495131 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:20:03.494792    2562 scope.go:117] "RemoveContainer" containerID="59829072a25fc4fb5b1fdba91fce084b0b101011d75ee759b750c241a888844e"
Sep 11 17:20:03.521918 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:20:03.521891    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d58fbcc7c-65k7r"]
Sep 11 17:20:03.528641 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:20:03.528617    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5d58fbcc7c-65k7r"]
Sep 11 17:20:04.124327 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:20:04.124295    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="025cbfc6-fdb8-41f9-b574-9c6accc6b576" path="/var/lib/kubelet/pods/025cbfc6-fdb8-41f9-b574-9c6accc6b576/volumes"
Sep 11 17:20:23.452858 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:20:23.452818    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-frlv9"
Sep 11 17:23:30.772786 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:30.772749    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b887fdd9f-pnsph"]
Sep 11 17:23:30.773208 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:30.773017    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="025cbfc6-fdb8-41f9-b574-9c6accc6b576" containerName="registry"
Sep 11 17:23:30.773208 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:30.773028    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="025cbfc6-fdb8-41f9-b574-9c6accc6b576" containerName="registry"
Sep 11 17:23:30.773208 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:30.773069    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="025cbfc6-fdb8-41f9-b574-9c6accc6b576" containerName="registry"
Sep 11 17:23:30.774797 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:30.774780    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-pnsph"
Sep 11 17:23:30.778069 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:30.778048    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 17:23:30.778197 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:30.778143    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 17:23:30.778252 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:30.778236    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 17:23:30.778398 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:30.778382    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 17:23:30.778463 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:30.778446    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-vxtdm\""
Sep 11 17:23:30.862743 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:30.862711    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br5qg\" (UniqueName: \"kubernetes.io/projected/4a57bf87-8a95-4d08-b354-14cc791deb5a-kube-api-access-br5qg\") pod \"opendatahub-operator-controller-manager-b887fdd9f-pnsph\" (UID: \"4a57bf87-8a95-4d08-b354-14cc791deb5a\") " pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-pnsph"
Sep 11 17:23:30.862743 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:30.862741    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a57bf87-8a95-4d08-b354-14cc791deb5a-tmp\") pod \"opendatahub-operator-controller-manager-b887fdd9f-pnsph\" (UID: \"4a57bf87-8a95-4d08-b354-14cc791deb5a\") " pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-pnsph"
Sep 11 17:23:30.862940 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:30.862769    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4a57bf87-8a95-4d08-b354-14cc791deb5a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b887fdd9f-pnsph\" (UID: \"4a57bf87-8a95-4d08-b354-14cc791deb5a\") " pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-pnsph"
Sep 11 17:23:30.862940 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:30.862860    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4a57bf87-8a95-4d08-b354-14cc791deb5a-manifests\") pod \"opendatahub-operator-controller-manager-b887fdd9f-pnsph\" (UID: \"4a57bf87-8a95-4d08-b354-14cc791deb5a\") " pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-pnsph"
Sep 11 17:23:30.862940 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:30.862897    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4a57bf87-8a95-4d08-b354-14cc791deb5a-webhook-cert\") pod \"opendatahub-operator-controller-manager-b887fdd9f-pnsph\" (UID: \"4a57bf87-8a95-4d08-b354-14cc791deb5a\") " pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-pnsph"
Sep 11 17:23:30.938866 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:30.938827    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b887fdd9f-pnsph"]
Sep 11 17:23:30.963775 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:30.963743    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4a57bf87-8a95-4d08-b354-14cc791deb5a-manifests\") pod \"opendatahub-operator-controller-manager-b887fdd9f-pnsph\" (UID: \"4a57bf87-8a95-4d08-b354-14cc791deb5a\") " pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-pnsph"
Sep 11 17:23:30.963936 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:30.963783    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4a57bf87-8a95-4d08-b354-14cc791deb5a-webhook-cert\") pod \"opendatahub-operator-controller-manager-b887fdd9f-pnsph\" (UID: \"4a57bf87-8a95-4d08-b354-14cc791deb5a\") " pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-pnsph"
Sep 11 17:23:30.963936 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:30.963809    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br5qg\" (UniqueName: \"kubernetes.io/projected/4a57bf87-8a95-4d08-b354-14cc791deb5a-kube-api-access-br5qg\") pod \"opendatahub-operator-controller-manager-b887fdd9f-pnsph\" (UID: \"4a57bf87-8a95-4d08-b354-14cc791deb5a\") " pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-pnsph"
Sep 11 17:23:30.963936 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:30.963826    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a57bf87-8a95-4d08-b354-14cc791deb5a-tmp\") pod \"opendatahub-operator-controller-manager-b887fdd9f-pnsph\" (UID: \"4a57bf87-8a95-4d08-b354-14cc791deb5a\") " pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-pnsph"
Sep 11 17:23:30.963936 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:30.963850    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4a57bf87-8a95-4d08-b354-14cc791deb5a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b887fdd9f-pnsph\" (UID: \"4a57bf87-8a95-4d08-b354-14cc791deb5a\") " pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-pnsph"
Sep 11 17:23:30.964190 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:30.964151    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4a57bf87-8a95-4d08-b354-14cc791deb5a-manifests\") pod \"opendatahub-operator-controller-manager-b887fdd9f-pnsph\" (UID: \"4a57bf87-8a95-4d08-b354-14cc791deb5a\") " pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-pnsph"
Sep 11 17:23:30.966276 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:30.966246    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a57bf87-8a95-4d08-b354-14cc791deb5a-tmp\") pod \"opendatahub-operator-controller-manager-b887fdd9f-pnsph\" (UID: \"4a57bf87-8a95-4d08-b354-14cc791deb5a\") " pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-pnsph"
Sep 11 17:23:30.966404 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:30.966284    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4a57bf87-8a95-4d08-b354-14cc791deb5a-webhook-cert\") pod \"opendatahub-operator-controller-manager-b887fdd9f-pnsph\" (UID: \"4a57bf87-8a95-4d08-b354-14cc791deb5a\") " pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-pnsph"
Sep 11 17:23:30.966404 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:30.966383    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4a57bf87-8a95-4d08-b354-14cc791deb5a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b887fdd9f-pnsph\" (UID: \"4a57bf87-8a95-4d08-b354-14cc791deb5a\") " pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-pnsph"
Sep 11 17:23:31.041461 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:31.041386    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-br5qg\" (UniqueName: \"kubernetes.io/projected/4a57bf87-8a95-4d08-b354-14cc791deb5a-kube-api-access-br5qg\") pod \"opendatahub-operator-controller-manager-b887fdd9f-pnsph\" (UID: \"4a57bf87-8a95-4d08-b354-14cc791deb5a\") " pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-pnsph"
Sep 11 17:23:31.084408 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:31.084371    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-pnsph"
Sep 11 17:23:31.386418 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:31.386387    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b887fdd9f-pnsph"]
Sep 11 17:23:31.390133 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:23:31.390106    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a57bf87_8a95_4d08_b354_14cc791deb5a.slice/crio-15682929032f73f7e72801400b8943039762b548a6f6556fd840d30e6b7d306e WatchSource:0}: Error finding container 15682929032f73f7e72801400b8943039762b548a6f6556fd840d30e6b7d306e: Status 404 returned error can't find the container with id 15682929032f73f7e72801400b8943039762b548a6f6556fd840d30e6b7d306e
Sep 11 17:23:32.154239 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:32.154201    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-pnsph" event={"ID":"4a57bf87-8a95-4d08-b354-14cc791deb5a","Type":"ContainerStarted","Data":"15682929032f73f7e72801400b8943039762b548a6f6556fd840d30e6b7d306e"}
Sep 11 17:23:35.167668 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:35.167576    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-pnsph" event={"ID":"4a57bf87-8a95-4d08-b354-14cc791deb5a","Type":"ContainerStarted","Data":"6527eaa209c4391ae8301d9a406890b2befd82c67f03665e6440fca352af4eac"}
Sep 11 17:23:36.172543 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:36.172508    2562 generic.go:356] "Generic (PLEG): container finished" podID="4a57bf87-8a95-4d08-b354-14cc791deb5a" containerID="6527eaa209c4391ae8301d9a406890b2befd82c67f03665e6440fca352af4eac" exitCode=0
Sep 11 17:23:36.172921 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:36.172573    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-pnsph" event={"ID":"4a57bf87-8a95-4d08-b354-14cc791deb5a","Type":"ContainerDied","Data":"6527eaa209c4391ae8301d9a406890b2befd82c67f03665e6440fca352af4eac"}
Sep 11 17:23:37.178743 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:37.178707    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-pnsph" event={"ID":"4a57bf87-8a95-4d08-b354-14cc791deb5a","Type":"ContainerStarted","Data":"2d690a63833249ac37a668a8e49dc660ff73d3167ba4a576bdecde5329cef431"}
Sep 11 17:23:37.179218 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:37.178965    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-pnsph"
Sep 11 17:23:37.207011 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:37.206944    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-pnsph" podStartSLOduration=2.23497468 podStartE2EDuration="7.206927066s" totalImagesPullingTime="3.891428601s" totalInitContainerRuntime="1.080523785s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:23:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:23:37.204492881 +0000 UTC m=+297.598410337" watchObservedRunningTime="2026-09-11 17:23:37.206927066 +0000 UTC m=+297.600844579"
Sep 11 17:23:40.002482 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:40.002457    2562 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 17:23:48.184735 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:48.184698    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-b887fdd9f-pnsph"
Sep 11 17:23:54.808583 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:54.808538    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7d6df6ff46-slct7"]
Sep 11 17:23:54.810674 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:54.810657    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d6df6ff46-slct7"
Sep 11 17:23:54.839338 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:23:54.839306    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-133-252.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-133-252.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Sep 11 17:23:54.847208 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:54.847180    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d6df6ff46-slct7"]
Sep 11 17:23:54.852628 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:23:54.852596    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kube-auth-proxy-7d6df6ff46-slct7\" is forbidden: User \"system:node:ip-10-0-133-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-133-252.ec2.internal' and this object" podUID="39f092d7-877a-4372-8f38-a32be1e7bd46" pod="openshift-ingress/kube-auth-proxy-7d6df6ff46-slct7"
Sep 11 17:23:54.853156 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:23:54.853134    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kube-auth-proxy-tls\" is forbidden: User \"system:node:ip-10-0-133-252.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-133-252.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" type="*v1.Secret"
Sep 11 17:23:54.853359 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:23:54.853342    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kube-auth-proxy-creds\" is forbidden: User \"system:node:ip-10-0-133-252.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-133-252.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" type="*v1.Secret"
Sep 11 17:23:54.853412 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:23:54.853351    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-133-252.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-133-252.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Sep 11 17:23:54.945137 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:54.945106    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39f092d7-877a-4372-8f38-a32be1e7bd46-tls-certs\") pod \"kube-auth-proxy-7d6df6ff46-slct7\" (UID: \"39f092d7-877a-4372-8f38-a32be1e7bd46\") " pod="openshift-ingress/kube-auth-proxy-7d6df6ff46-slct7"
Sep 11 17:23:54.945314 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:54.945151    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g2sq\" (UniqueName: \"kubernetes.io/projected/39f092d7-877a-4372-8f38-a32be1e7bd46-kube-api-access-9g2sq\") pod \"kube-auth-proxy-7d6df6ff46-slct7\" (UID: \"39f092d7-877a-4372-8f38-a32be1e7bd46\") " pod="openshift-ingress/kube-auth-proxy-7d6df6ff46-slct7"
Sep 11 17:23:54.945314 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:54.945182    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39f092d7-877a-4372-8f38-a32be1e7bd46-tmp\") pod \"kube-auth-proxy-7d6df6ff46-slct7\" (UID: \"39f092d7-877a-4372-8f38-a32be1e7bd46\") " pod="openshift-ingress/kube-auth-proxy-7d6df6ff46-slct7"
Sep 11 17:23:55.046114 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:55.046073    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39f092d7-877a-4372-8f38-a32be1e7bd46-tls-certs\") pod \"kube-auth-proxy-7d6df6ff46-slct7\" (UID: \"39f092d7-877a-4372-8f38-a32be1e7bd46\") " pod="openshift-ingress/kube-auth-proxy-7d6df6ff46-slct7"
Sep 11 17:23:55.046319 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:55.046161    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g2sq\" (UniqueName: \"kubernetes.io/projected/39f092d7-877a-4372-8f38-a32be1e7bd46-kube-api-access-9g2sq\") pod \"kube-auth-proxy-7d6df6ff46-slct7\" (UID: \"39f092d7-877a-4372-8f38-a32be1e7bd46\") " pod="openshift-ingress/kube-auth-proxy-7d6df6ff46-slct7"
Sep 11 17:23:55.046319 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:55.046226    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39f092d7-877a-4372-8f38-a32be1e7bd46-tmp\") pod \"kube-auth-proxy-7d6df6ff46-slct7\" (UID: \"39f092d7-877a-4372-8f38-a32be1e7bd46\") " pod="openshift-ingress/kube-auth-proxy-7d6df6ff46-slct7"
Sep 11 17:23:55.048594 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:55.048558    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39f092d7-877a-4372-8f38-a32be1e7bd46-tmp\") pod \"kube-auth-proxy-7d6df6ff46-slct7\" (UID: \"39f092d7-877a-4372-8f38-a32be1e7bd46\") " pod="openshift-ingress/kube-auth-proxy-7d6df6ff46-slct7"
Sep 11 17:23:55.884526 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:55.884493    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 11 17:23:55.888822 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:55.888794    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39f092d7-877a-4372-8f38-a32be1e7bd46-tls-certs\") pod \"kube-auth-proxy-7d6df6ff46-slct7\" (UID: \"39f092d7-877a-4372-8f38-a32be1e7bd46\") " pod="openshift-ingress/kube-auth-proxy-7d6df6ff46-slct7"
Sep 11 17:23:56.060415 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:23:56.060382    2562 projected.go:291] Couldn't get configMap openshift-ingress/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition
Sep 11 17:23:56.129135 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:56.129105    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 11 17:23:56.369066 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:56.369038    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 17:23:56.370912 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:23:56.370893    2562 projected.go:196] Error preparing data for projected volume kube-api-access-9g2sq for pod openshift-ingress/kube-auth-proxy-7d6df6ff46-slct7: failed to sync configmap cache: timed out waiting for the condition
Sep 11 17:23:56.371032 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:23:56.370967    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39f092d7-877a-4372-8f38-a32be1e7bd46-kube-api-access-9g2sq podName:39f092d7-877a-4372-8f38-a32be1e7bd46 nodeName:}" failed. No retries permitted until 2026-09-11 17:23:56.870945576 +0000 UTC m=+317.264863028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9g2sq" (UniqueName: "kubernetes.io/projected/39f092d7-877a-4372-8f38-a32be1e7bd46-kube-api-access-9g2sq") pod "kube-auth-proxy-7d6df6ff46-slct7" (UID: "39f092d7-877a-4372-8f38-a32be1e7bd46") : failed to sync configmap cache: timed out waiting for the condition
Sep 11 17:23:56.394926 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:56.394902    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 17:23:56.866704 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:56.866672    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw"]
Sep 11 17:23:56.868906 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:56.868890    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw"
Sep 11 17:23:56.871211 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:56.871188    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 11 17:23:56.871430 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:56.871412    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 11 17:23:56.871656 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:56.871631    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-ttt5w\""
Sep 11 17:23:56.871762 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:56.871669    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 11 17:23:56.871866 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:56.871846    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 17:23:56.886698 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:56.886674    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw"]
Sep 11 17:23:56.962007 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:56.961969    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0d898dc9-91b1-49f9-9451-29651b917824-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-cfnvw\" (UID: \"0d898dc9-91b1-49f9-9451-29651b917824\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw"
Sep 11 17:23:56.962178 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:56.962030    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0d898dc9-91b1-49f9-9451-29651b917824-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-cfnvw\" (UID: \"0d898dc9-91b1-49f9-9451-29651b917824\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw"
Sep 11 17:23:56.962178 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:56.962111    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0d898dc9-91b1-49f9-9451-29651b917824-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-cfnvw\" (UID: \"0d898dc9-91b1-49f9-9451-29651b917824\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw"
Sep 11 17:23:56.962178 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:56.962146    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d898dc9-91b1-49f9-9451-29651b917824-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-cfnvw\" (UID: \"0d898dc9-91b1-49f9-9451-29651b917824\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw"
Sep 11 17:23:56.962343 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:56.962234    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d898dc9-91b1-49f9-9451-29651b917824-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-cfnvw\" (UID: \"0d898dc9-91b1-49f9-9451-29651b917824\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw"
Sep 11 17:23:56.962343 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:56.962288    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g2sq\" (UniqueName: \"kubernetes.io/projected/39f092d7-877a-4372-8f38-a32be1e7bd46-kube-api-access-9g2sq\") pod \"kube-auth-proxy-7d6df6ff46-slct7\" (UID: \"39f092d7-877a-4372-8f38-a32be1e7bd46\") " pod="openshift-ingress/kube-auth-proxy-7d6df6ff46-slct7"
Sep 11 17:23:56.962343 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:56.962321    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0d898dc9-91b1-49f9-9451-29651b917824-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-cfnvw\" (UID: \"0d898dc9-91b1-49f9-9451-29651b917824\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw"
Sep 11 17:23:56.962476 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:56.962372    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnrqd\" (UniqueName: \"kubernetes.io/projected/0d898dc9-91b1-49f9-9451-29651b917824-kube-api-access-vnrqd\") pod \"istiod-openshift-gateway-7b7995d5cd-cfnvw\" (UID: \"0d898dc9-91b1-49f9-9451-29651b917824\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw"
Sep 11 17:23:56.964741 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:56.964724    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g2sq\" (UniqueName: \"kubernetes.io/projected/39f092d7-877a-4372-8f38-a32be1e7bd46-kube-api-access-9g2sq\") pod \"kube-auth-proxy-7d6df6ff46-slct7\" (UID: \"39f092d7-877a-4372-8f38-a32be1e7bd46\") " pod="openshift-ingress/kube-auth-proxy-7d6df6ff46-slct7"
Sep 11 17:23:57.063519 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:57.063466    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnrqd\" (UniqueName: \"kubernetes.io/projected/0d898dc9-91b1-49f9-9451-29651b917824-kube-api-access-vnrqd\") pod \"istiod-openshift-gateway-7b7995d5cd-cfnvw\" (UID: \"0d898dc9-91b1-49f9-9451-29651b917824\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw"
Sep 11 17:23:57.063519 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:57.063519    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0d898dc9-91b1-49f9-9451-29651b917824-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-cfnvw\" (UID: \"0d898dc9-91b1-49f9-9451-29651b917824\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw"
Sep 11 17:23:57.063778 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:57.063543    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0d898dc9-91b1-49f9-9451-29651b917824-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-cfnvw\" (UID: \"0d898dc9-91b1-49f9-9451-29651b917824\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw"
Sep 11 17:23:57.063778 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:57.063566    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0d898dc9-91b1-49f9-9451-29651b917824-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-cfnvw\" (UID: \"0d898dc9-91b1-49f9-9451-29651b917824\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw"
Sep 11 17:23:57.063778 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:57.063586    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d898dc9-91b1-49f9-9451-29651b917824-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-cfnvw\" (UID: \"0d898dc9-91b1-49f9-9451-29651b917824\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw"
Sep 11 17:23:57.063778 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:57.063608    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d898dc9-91b1-49f9-9451-29651b917824-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-cfnvw\" (UID: \"0d898dc9-91b1-49f9-9451-29651b917824\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw"
Sep 11 17:23:57.063778 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:57.063634    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0d898dc9-91b1-49f9-9451-29651b917824-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-cfnvw\" (UID: \"0d898dc9-91b1-49f9-9451-29651b917824\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw"
Sep 11 17:23:57.064339 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:57.064316    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0d898dc9-91b1-49f9-9451-29651b917824-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-cfnvw\" (UID: \"0d898dc9-91b1-49f9-9451-29651b917824\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw"
Sep 11 17:23:57.065937 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:57.065915    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0d898dc9-91b1-49f9-9451-29651b917824-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-cfnvw\" (UID: \"0d898dc9-91b1-49f9-9451-29651b917824\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw"
Sep 11 17:23:57.066046 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:57.065956    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0d898dc9-91b1-49f9-9451-29651b917824-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-cfnvw\" (UID: \"0d898dc9-91b1-49f9-9451-29651b917824\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw"
Sep 11 17:23:57.066272 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:57.066248    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d898dc9-91b1-49f9-9451-29651b917824-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-cfnvw\" (UID: \"0d898dc9-91b1-49f9-9451-29651b917824\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw"
Sep 11 17:23:57.066330 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:57.066300    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0d898dc9-91b1-49f9-9451-29651b917824-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-cfnvw\" (UID: \"0d898dc9-91b1-49f9-9451-29651b917824\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw"
Sep 11 17:23:57.071752 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:57.071727    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnrqd\" (UniqueName: \"kubernetes.io/projected/0d898dc9-91b1-49f9-9451-29651b917824-kube-api-access-vnrqd\") pod \"istiod-openshift-gateway-7b7995d5cd-cfnvw\" (UID: \"0d898dc9-91b1-49f9-9451-29651b917824\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw"
Sep 11 17:23:57.072915 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:57.072896    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d898dc9-91b1-49f9-9451-29651b917824-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-cfnvw\" (UID: \"0d898dc9-91b1-49f9-9451-29651b917824\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw"
Sep 11 17:23:57.178252 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:57.178160    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw"
Sep 11 17:23:57.222088 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:57.222056    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d6df6ff46-slct7"
Sep 11 17:23:57.458628 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:57.458548    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw"]
Sep 11 17:23:57.461802 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:23:57.461778    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d898dc9_91b1_49f9_9451_29651b917824.slice/crio-66cb748bdcc85c71979b276b79d9a27222cbbda981660d4848eb0f14e3245488 WatchSource:0}: Error finding container 66cb748bdcc85c71979b276b79d9a27222cbbda981660d4848eb0f14e3245488: Status 404 returned error can't find the container with id 66cb748bdcc85c71979b276b79d9a27222cbbda981660d4848eb0f14e3245488
Sep 11 17:23:57.463768 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:57.463751    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 17:23:57.507495 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:57.507471    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d6df6ff46-slct7"]
Sep 11 17:23:57.509808 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:23:57.509783    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39f092d7_877a_4372_8f38_a32be1e7bd46.slice/crio-da62898d75472ba177608ac2446aa4fe362e75ee4307637f502cc45d81985b77 WatchSource:0}: Error finding container da62898d75472ba177608ac2446aa4fe362e75ee4307637f502cc45d81985b77: Status 404 returned error can't find the container with id da62898d75472ba177608ac2446aa4fe362e75ee4307637f502cc45d81985b77
Sep 11 17:23:58.253808 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:58.253721    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d6df6ff46-slct7" event={"ID":"39f092d7-877a-4372-8f38-a32be1e7bd46","Type":"ContainerStarted","Data":"da62898d75472ba177608ac2446aa4fe362e75ee4307637f502cc45d81985b77"}
Sep 11 17:23:58.256429 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:23:58.256362    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw" event={"ID":"0d898dc9-91b1-49f9-9451-29651b917824","Type":"ContainerStarted","Data":"66cb748bdcc85c71979b276b79d9a27222cbbda981660d4848eb0f14e3245488"}
Sep 11 17:24:01.139102 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:01.139062    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:24:01.139498 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:01.139155    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:24:01.268983 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:01.268943    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw" event={"ID":"0d898dc9-91b1-49f9-9451-29651b917824","Type":"ContainerStarted","Data":"025af7f5ca225d24564d8e1a585eaae647236594574493938eee4224cbff83ac"}
Sep 11 17:24:01.269187 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:01.269027    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw"
Sep 11 17:24:01.270576 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:01.270548    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d6df6ff46-slct7" event={"ID":"39f092d7-877a-4372-8f38-a32be1e7bd46","Type":"ContainerStarted","Data":"2732da1e0221bd3e6b7fc40abcef3c3e55aa239bc679f0270c7151da91144337"}
Sep 11 17:24:01.291890 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:01.291839    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw" podStartSLOduration=1.616873842 podStartE2EDuration="5.291823819s" totalImagesPullingTime="3.674949977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:23:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:24:01.290453891 +0000 UTC m=+321.684371346" watchObservedRunningTime="2026-09-11 17:24:01.291823819 +0000 UTC m=+321.685741274"
Sep 11 17:24:01.314827 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:01.314777    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7d6df6ff46-slct7" podStartSLOduration=3.6849303129999997 podStartE2EDuration="7.314759101s" totalImagesPullingTime="3.629828788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:23:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:24:01.314522301 +0000 UTC m=+321.708439756" watchObservedRunningTime="2026-09-11 17:24:01.314759101 +0000 UTC m=+321.708676558"
Sep 11 17:24:02.277301 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:02.277264    2562 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-cfnvw container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 17:24:02.277776 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:02.277326    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw" podUID="0d898dc9-91b1-49f9-9451-29651b917824" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 17:24:05.277203 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:05.277172    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw"
Sep 11 17:24:09.802711 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:09.802670    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-rp5zl"]
Sep 11 17:24:09.806399 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:09.806380    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rp5zl"
Sep 11 17:24:09.852847 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:09.852815    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-rp5zl"]
Sep 11 17:24:09.968475 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:09.968443    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f26863e-47a2-44f5-9ba9-6e7643bbf847-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-rp5zl\" (UID: \"9f26863e-47a2-44f5-9ba9-6e7643bbf847\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rp5zl"
Sep 11 17:24:09.968475 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:09.968482    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f26863e-47a2-44f5-9ba9-6e7643bbf847-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-rp5zl\" (UID: \"9f26863e-47a2-44f5-9ba9-6e7643bbf847\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rp5zl"
Sep 11 17:24:09.968673 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:09.968500    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9f26863e-47a2-44f5-9ba9-6e7643bbf847-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-rp5zl\" (UID: \"9f26863e-47a2-44f5-9ba9-6e7643bbf847\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rp5zl"
Sep 11 17:24:09.968673 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:09.968605    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9f26863e-47a2-44f5-9ba9-6e7643bbf847-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-rp5zl\" (UID: \"9f26863e-47a2-44f5-9ba9-6e7643bbf847\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rp5zl"
Sep 11 17:24:09.968673 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:09.968635    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7j58\" (UniqueName: \"kubernetes.io/projected/9f26863e-47a2-44f5-9ba9-6e7643bbf847-kube-api-access-g7j58\") pod \"istiod-openshift-gateway-5677ccb49b-rp5zl\" (UID: \"9f26863e-47a2-44f5-9ba9-6e7643bbf847\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rp5zl"
Sep 11 17:24:09.968673 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:09.968662    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9f26863e-47a2-44f5-9ba9-6e7643bbf847-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-rp5zl\" (UID: \"9f26863e-47a2-44f5-9ba9-6e7643bbf847\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rp5zl"
Sep 11 17:24:09.968798 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:09.968680    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f26863e-47a2-44f5-9ba9-6e7643bbf847-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-rp5zl\" (UID: \"9f26863e-47a2-44f5-9ba9-6e7643bbf847\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rp5zl"
Sep 11 17:24:10.069483 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:10.069361    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9f26863e-47a2-44f5-9ba9-6e7643bbf847-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-rp5zl\" (UID: \"9f26863e-47a2-44f5-9ba9-6e7643bbf847\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rp5zl"
Sep 11 17:24:10.069483 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:10.069416    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7j58\" (UniqueName: \"kubernetes.io/projected/9f26863e-47a2-44f5-9ba9-6e7643bbf847-kube-api-access-g7j58\") pod \"istiod-openshift-gateway-5677ccb49b-rp5zl\" (UID: \"9f26863e-47a2-44f5-9ba9-6e7643bbf847\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rp5zl"
Sep 11 17:24:10.069483 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:10.069468    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9f26863e-47a2-44f5-9ba9-6e7643bbf847-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-rp5zl\" (UID: \"9f26863e-47a2-44f5-9ba9-6e7643bbf847\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rp5zl"
Sep 11 17:24:10.069760 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:10.069497    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f26863e-47a2-44f5-9ba9-6e7643bbf847-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-rp5zl\" (UID: \"9f26863e-47a2-44f5-9ba9-6e7643bbf847\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rp5zl"
Sep 11 17:24:10.069760 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:10.069534    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f26863e-47a2-44f5-9ba9-6e7643bbf847-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-rp5zl\" (UID: \"9f26863e-47a2-44f5-9ba9-6e7643bbf847\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rp5zl"
Sep 11 17:24:10.069760 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:10.069566    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f26863e-47a2-44f5-9ba9-6e7643bbf847-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-rp5zl\" (UID: \"9f26863e-47a2-44f5-9ba9-6e7643bbf847\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rp5zl"
Sep 11 17:24:10.069760 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:10.069593    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9f26863e-47a2-44f5-9ba9-6e7643bbf847-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-rp5zl\" (UID: \"9f26863e-47a2-44f5-9ba9-6e7643bbf847\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rp5zl"
Sep 11 17:24:10.070262 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:10.070229    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9f26863e-47a2-44f5-9ba9-6e7643bbf847-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-rp5zl\" (UID: \"9f26863e-47a2-44f5-9ba9-6e7643bbf847\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rp5zl"
Sep 11 17:24:10.072046 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:10.072021    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f26863e-47a2-44f5-9ba9-6e7643bbf847-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-rp5zl\" (UID: \"9f26863e-47a2-44f5-9ba9-6e7643bbf847\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rp5zl"
Sep 11 17:24:10.072406 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:10.072386    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f26863e-47a2-44f5-9ba9-6e7643bbf847-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-rp5zl\" (UID: \"9f26863e-47a2-44f5-9ba9-6e7643bbf847\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rp5zl"
Sep 11 17:24:10.072486 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:10.072437    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9f26863e-47a2-44f5-9ba9-6e7643bbf847-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-rp5zl\" (UID: \"9f26863e-47a2-44f5-9ba9-6e7643bbf847\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rp5zl"
Sep 11 17:24:10.072486 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:10.072478    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9f26863e-47a2-44f5-9ba9-6e7643bbf847-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-rp5zl\" (UID: \"9f26863e-47a2-44f5-9ba9-6e7643bbf847\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rp5zl"
Sep 11 17:24:10.083986 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:10.083959    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7j58\" (UniqueName: \"kubernetes.io/projected/9f26863e-47a2-44f5-9ba9-6e7643bbf847-kube-api-access-g7j58\") pod \"istiod-openshift-gateway-5677ccb49b-rp5zl\" (UID: \"9f26863e-47a2-44f5-9ba9-6e7643bbf847\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rp5zl"
Sep 11 17:24:10.087901 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:10.087876    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f26863e-47a2-44f5-9ba9-6e7643bbf847-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-rp5zl\" (UID: \"9f26863e-47a2-44f5-9ba9-6e7643bbf847\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rp5zl"
Sep 11 17:24:10.115150 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:10.115123    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rp5zl"
Sep 11 17:24:10.409255 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:10.409233    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-rp5zl"]
Sep 11 17:24:10.411460 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:24:10.411433    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f26863e_47a2_44f5_9ba9_6e7643bbf847.slice/crio-0cf9719e0f806fc882a14817fe645420363f7d89c942525696980f3570823759 WatchSource:0}: Error finding container 0cf9719e0f806fc882a14817fe645420363f7d89c942525696980f3570823759: Status 404 returned error can't find the container with id 0cf9719e0f806fc882a14817fe645420363f7d89c942525696980f3570823759
Sep 11 17:24:10.414518 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:10.414479    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:24:10.414637 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:10.414546    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:24:11.311023 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:11.310968    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rp5zl" event={"ID":"9f26863e-47a2-44f5-9ba9-6e7643bbf847","Type":"ContainerStarted","Data":"2293b98db6257e8fe7e5e52cbae31855a7191b745c270e121fce547e9ecbc31b"}
Sep 11 17:24:11.311506 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:11.311044    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rp5zl" event={"ID":"9f26863e-47a2-44f5-9ba9-6e7643bbf847","Type":"ContainerStarted","Data":"0cf9719e0f806fc882a14817fe645420363f7d89c942525696980f3570823759"}
Sep 11 17:24:11.311506 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:11.311162    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rp5zl"
Sep 11 17:24:11.312934 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:11.312912    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rp5zl"
Sep 11 17:24:11.357013 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:11.353873    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rp5zl" podStartSLOduration=2.353851676 podStartE2EDuration="2.353851676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:24:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:24:11.350432284 +0000 UTC m=+331.744349739" watchObservedRunningTime="2026-09-11 17:24:11.353851676 +0000 UTC m=+331.747769132"
Sep 11 17:24:11.468618 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:11.468586    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw"]
Sep 11 17:24:11.468839 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:11.468805    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw" podUID="0d898dc9-91b1-49f9-9451-29651b917824" containerName="discovery" containerID="cri-o://025af7f5ca225d24564d8e1a585eaae647236594574493938eee4224cbff83ac" gracePeriod=30
Sep 11 17:24:11.705194 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:11.705159    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw"
Sep 11 17:24:11.784632 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:11.784600    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0d898dc9-91b1-49f9-9451-29651b917824-istio-csr-ca-configmap\") pod \"0d898dc9-91b1-49f9-9451-29651b917824\" (UID: \"0d898dc9-91b1-49f9-9451-29651b917824\") "
Sep 11 17:24:11.784799 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:11.784639    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d898dc9-91b1-49f9-9451-29651b917824-istio-token\") pod \"0d898dc9-91b1-49f9-9451-29651b917824\" (UID: \"0d898dc9-91b1-49f9-9451-29651b917824\") "
Sep 11 17:24:11.784799 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:11.784669    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0d898dc9-91b1-49f9-9451-29651b917824-istio-kubeconfig\") pod \"0d898dc9-91b1-49f9-9451-29651b917824\" (UID: \"0d898dc9-91b1-49f9-9451-29651b917824\") "
Sep 11 17:24:11.784799 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:11.784689    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vnrqd\" (UniqueName: \"kubernetes.io/projected/0d898dc9-91b1-49f9-9451-29651b917824-kube-api-access-vnrqd\") pod \"0d898dc9-91b1-49f9-9451-29651b917824\" (UID: \"0d898dc9-91b1-49f9-9451-29651b917824\") "
Sep 11 17:24:11.784799 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:11.784712    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0d898dc9-91b1-49f9-9451-29651b917824-istio-csr-dns-cert\") pod \"0d898dc9-91b1-49f9-9451-29651b917824\" (UID: \"0d898dc9-91b1-49f9-9451-29651b917824\") "
Sep 11 17:24:11.784799 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:11.784744    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0d898dc9-91b1-49f9-9451-29651b917824-cacerts\") pod \"0d898dc9-91b1-49f9-9451-29651b917824\" (UID: \"0d898dc9-91b1-49f9-9451-29651b917824\") "
Sep 11 17:24:11.785167 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:11.785026    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d898dc9-91b1-49f9-9451-29651b917824-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "0d898dc9-91b1-49f9-9451-29651b917824" (UID: "0d898dc9-91b1-49f9-9451-29651b917824"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:24:11.785167 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:11.785133    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d898dc9-91b1-49f9-9451-29651b917824-local-certs\") pod \"0d898dc9-91b1-49f9-9451-29651b917824\" (UID: \"0d898dc9-91b1-49f9-9451-29651b917824\") "
Sep 11 17:24:11.785448 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:11.785411    2562 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0d898dc9-91b1-49f9-9451-29651b917824-istio-csr-ca-configmap\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:24:11.787172 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:11.787140    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d898dc9-91b1-49f9-9451-29651b917824-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "0d898dc9-91b1-49f9-9451-29651b917824" (UID: "0d898dc9-91b1-49f9-9451-29651b917824"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:24:11.787172 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:11.787162    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d898dc9-91b1-49f9-9451-29651b917824-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "0d898dc9-91b1-49f9-9451-29651b917824" (UID: "0d898dc9-91b1-49f9-9451-29651b917824"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:24:11.787348 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:11.787331    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d898dc9-91b1-49f9-9451-29651b917824-kube-api-access-vnrqd" (OuterVolumeSpecName: "kube-api-access-vnrqd") pod "0d898dc9-91b1-49f9-9451-29651b917824" (UID: "0d898dc9-91b1-49f9-9451-29651b917824"). InnerVolumeSpecName "kube-api-access-vnrqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:24:11.787348 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:11.787341    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d898dc9-91b1-49f9-9451-29651b917824-cacerts" (OuterVolumeSpecName: "cacerts") pod "0d898dc9-91b1-49f9-9451-29651b917824" (UID: "0d898dc9-91b1-49f9-9451-29651b917824"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:24:11.787451 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:11.787427    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d898dc9-91b1-49f9-9451-29651b917824-istio-token" (OuterVolumeSpecName: "istio-token") pod "0d898dc9-91b1-49f9-9451-29651b917824" (UID: "0d898dc9-91b1-49f9-9451-29651b917824"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:24:11.787509 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:11.787457    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d898dc9-91b1-49f9-9451-29651b917824-local-certs" (OuterVolumeSpecName: "local-certs") pod "0d898dc9-91b1-49f9-9451-29651b917824" (UID: "0d898dc9-91b1-49f9-9451-29651b917824"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:24:11.885781 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:11.885743    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d898dc9-91b1-49f9-9451-29651b917824-istio-token\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:24:11.885781 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:11.885772    2562 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0d898dc9-91b1-49f9-9451-29651b917824-istio-kubeconfig\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:24:11.885781 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:11.885786    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnrqd\" (UniqueName: \"kubernetes.io/projected/0d898dc9-91b1-49f9-9451-29651b917824-kube-api-access-vnrqd\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:24:11.886065 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:11.885799    2562 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0d898dc9-91b1-49f9-9451-29651b917824-istio-csr-dns-cert\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:24:11.886065 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:11.885809    2562 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0d898dc9-91b1-49f9-9451-29651b917824-cacerts\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:24:11.886065 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:11.885821    2562 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d898dc9-91b1-49f9-9451-29651b917824-local-certs\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:24:12.316264 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:12.316175    2562 generic.go:356] "Generic (PLEG): container finished" podID="0d898dc9-91b1-49f9-9451-29651b917824" containerID="025af7f5ca225d24564d8e1a585eaae647236594574493938eee4224cbff83ac" exitCode=0
Sep 11 17:24:12.316264 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:12.316230    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw"
Sep 11 17:24:12.316769 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:12.316267    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw" event={"ID":"0d898dc9-91b1-49f9-9451-29651b917824","Type":"ContainerDied","Data":"025af7f5ca225d24564d8e1a585eaae647236594574493938eee4224cbff83ac"}
Sep 11 17:24:12.316769 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:12.316317    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw" event={"ID":"0d898dc9-91b1-49f9-9451-29651b917824","Type":"ContainerDied","Data":"66cb748bdcc85c71979b276b79d9a27222cbbda981660d4848eb0f14e3245488"}
Sep 11 17:24:12.316769 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:12.316338    2562 scope.go:117] "RemoveContainer" containerID="025af7f5ca225d24564d8e1a585eaae647236594574493938eee4224cbff83ac"
Sep 11 17:24:12.324479 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:12.324450    2562 scope.go:117] "RemoveContainer" containerID="025af7f5ca225d24564d8e1a585eaae647236594574493938eee4224cbff83ac"
Sep 11 17:24:12.324809 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:24:12.324776    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"025af7f5ca225d24564d8e1a585eaae647236594574493938eee4224cbff83ac\": container with ID starting with 025af7f5ca225d24564d8e1a585eaae647236594574493938eee4224cbff83ac not found: ID does not exist" containerID="025af7f5ca225d24564d8e1a585eaae647236594574493938eee4224cbff83ac"
Sep 11 17:24:12.324904 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:12.324820    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"025af7f5ca225d24564d8e1a585eaae647236594574493938eee4224cbff83ac"} err="failed to get container status \"025af7f5ca225d24564d8e1a585eaae647236594574493938eee4224cbff83ac\": rpc error: code = NotFound desc = could not find container \"025af7f5ca225d24564d8e1a585eaae647236594574493938eee4224cbff83ac\": container with ID starting with 025af7f5ca225d24564d8e1a585eaae647236594574493938eee4224cbff83ac not found: ID does not exist"
Sep 11 17:24:12.363810 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:12.363778    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw"]
Sep 11 17:24:12.387880 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:12.387852    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfnvw"]
Sep 11 17:24:13.146828 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:13.146797    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-8bbvh"]
Sep 11 17:24:13.147106 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:13.147092    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d898dc9-91b1-49f9-9451-29651b917824" containerName="discovery"
Sep 11 17:24:13.147172 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:13.147108    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d898dc9-91b1-49f9-9451-29651b917824" containerName="discovery"
Sep 11 17:24:13.147172 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:13.147147    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d898dc9-91b1-49f9-9451-29651b917824" containerName="discovery"
Sep 11 17:24:13.151243 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:13.151219    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8bbvh"
Sep 11 17:24:13.155386 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:13.155366    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-9zx9w\""
Sep 11 17:24:13.155509 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:13.155406    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 11 17:24:13.184060 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:13.184017    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-8bbvh"]
Sep 11 17:24:13.296825 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:13.296791    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvpht\" (UniqueName: \"kubernetes.io/projected/eca3874b-6600-44d9-99a3-4ee52e6e1c6b-kube-api-access-zvpht\") pod \"kserve-controller-manager-d6f4b65b8-8bbvh\" (UID: \"eca3874b-6600-44d9-99a3-4ee52e6e1c6b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8bbvh"
Sep 11 17:24:13.297029 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:13.296849    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eca3874b-6600-44d9-99a3-4ee52e6e1c6b-cert\") pod \"kserve-controller-manager-d6f4b65b8-8bbvh\" (UID: \"eca3874b-6600-44d9-99a3-4ee52e6e1c6b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8bbvh"
Sep 11 17:24:13.397515 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:13.397420    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvpht\" (UniqueName: \"kubernetes.io/projected/eca3874b-6600-44d9-99a3-4ee52e6e1c6b-kube-api-access-zvpht\") pod \"kserve-controller-manager-d6f4b65b8-8bbvh\" (UID: \"eca3874b-6600-44d9-99a3-4ee52e6e1c6b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8bbvh"
Sep 11 17:24:13.397515 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:13.397474    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eca3874b-6600-44d9-99a3-4ee52e6e1c6b-cert\") pod \"kserve-controller-manager-d6f4b65b8-8bbvh\" (UID: \"eca3874b-6600-44d9-99a3-4ee52e6e1c6b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8bbvh"
Sep 11 17:24:13.399833 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:13.399807    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eca3874b-6600-44d9-99a3-4ee52e6e1c6b-cert\") pod \"kserve-controller-manager-d6f4b65b8-8bbvh\" (UID: \"eca3874b-6600-44d9-99a3-4ee52e6e1c6b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8bbvh"
Sep 11 17:24:13.409312 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:13.409290    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvpht\" (UniqueName: \"kubernetes.io/projected/eca3874b-6600-44d9-99a3-4ee52e6e1c6b-kube-api-access-zvpht\") pod \"kserve-controller-manager-d6f4b65b8-8bbvh\" (UID: \"eca3874b-6600-44d9-99a3-4ee52e6e1c6b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8bbvh"
Sep 11 17:24:13.461139 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:13.461109    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8bbvh"
Sep 11 17:24:13.748231 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:13.748195    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-8bbvh"]
Sep 11 17:24:13.748571 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:24:13.748548    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeca3874b_6600_44d9_99a3_4ee52e6e1c6b.slice/crio-45ee011a4ccd08e741fd0a47cef853aee44774fe0230e18fcd05d8aba42501fe WatchSource:0}: Error finding container 45ee011a4ccd08e741fd0a47cef853aee44774fe0230e18fcd05d8aba42501fe: Status 404 returned error can't find the container with id 45ee011a4ccd08e741fd0a47cef853aee44774fe0230e18fcd05d8aba42501fe
Sep 11 17:24:14.123736 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:14.123700    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d898dc9-91b1-49f9-9451-29651b917824" path="/var/lib/kubelet/pods/0d898dc9-91b1-49f9-9451-29651b917824/volumes"
Sep 11 17:24:14.326311 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:14.326278    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8bbvh" event={"ID":"eca3874b-6600-44d9-99a3-4ee52e6e1c6b","Type":"ContainerStarted","Data":"45ee011a4ccd08e741fd0a47cef853aee44774fe0230e18fcd05d8aba42501fe"}
Sep 11 17:24:16.335525 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:16.335436    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8bbvh" event={"ID":"eca3874b-6600-44d9-99a3-4ee52e6e1c6b","Type":"ContainerStarted","Data":"5e88b14027424271d7a34570e729abebb0b63dd795c248de756aedab5ce8cda8"}
Sep 11 17:24:16.335895 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:16.335571    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8bbvh"
Sep 11 17:24:16.371805 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:16.371757    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8bbvh" podStartSLOduration=1.20737621 podStartE2EDuration="3.371741308s" totalImagesPullingTime="2.164365098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:24:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:24:16.371153638 +0000 UTC m=+336.765071094" watchObservedRunningTime="2026-09-11 17:24:16.371741308 +0000 UTC m=+336.765658763"
Sep 11 17:24:47.344860 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:24:47.344826    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8bbvh"
Sep 11 17:25:24.593310 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:24.593278    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86kc8"]
Sep 11 17:25:24.595571 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:24.595556    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86kc8"
Sep 11 17:25:24.598808 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:24.598783    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 17:25:24.598948 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:24.598889    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 17:25:24.601354 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:24.601337    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-dztsz\""
Sep 11 17:25:24.650403 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:24.650374    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86kc8"]
Sep 11 17:25:24.754643 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:24.754608    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea300ea1-39df-4925-add3-13a8f3823b67-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-86kc8\" (UID: \"ea300ea1-39df-4925-add3-13a8f3823b67\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86kc8"
Sep 11 17:25:24.754643 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:24.754645    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mggj\" (UniqueName: \"kubernetes.io/projected/ea300ea1-39df-4925-add3-13a8f3823b67-kube-api-access-7mggj\") pod \"kuadrant-operator-controller-manager-76f8f686bf-86kc8\" (UID: \"ea300ea1-39df-4925-add3-13a8f3823b67\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86kc8"
Sep 11 17:25:24.856061 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:24.855961    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea300ea1-39df-4925-add3-13a8f3823b67-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-86kc8\" (UID: \"ea300ea1-39df-4925-add3-13a8f3823b67\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86kc8"
Sep 11 17:25:24.856061 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:24.856016    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mggj\" (UniqueName: \"kubernetes.io/projected/ea300ea1-39df-4925-add3-13a8f3823b67-kube-api-access-7mggj\") pod \"kuadrant-operator-controller-manager-76f8f686bf-86kc8\" (UID: \"ea300ea1-39df-4925-add3-13a8f3823b67\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86kc8"
Sep 11 17:25:24.856360 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:24.856343    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea300ea1-39df-4925-add3-13a8f3823b67-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-86kc8\" (UID: \"ea300ea1-39df-4925-add3-13a8f3823b67\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86kc8"
Sep 11 17:25:24.874349 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:24.874326    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mggj\" (UniqueName: \"kubernetes.io/projected/ea300ea1-39df-4925-add3-13a8f3823b67-kube-api-access-7mggj\") pod \"kuadrant-operator-controller-manager-76f8f686bf-86kc8\" (UID: \"ea300ea1-39df-4925-add3-13a8f3823b67\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86kc8"
Sep 11 17:25:24.905310 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:24.905286    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86kc8"
Sep 11 17:25:25.188587 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:25.188489    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86kc8"]
Sep 11 17:25:25.192811 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:25:25.192782    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea300ea1_39df_4925_add3_13a8f3823b67.slice/crio-53dba0e1b68733772438c122094e6fc876efa56623b28c26a957ad003ba1885f WatchSource:0}: Error finding container 53dba0e1b68733772438c122094e6fc876efa56623b28c26a957ad003ba1885f: Status 404 returned error can't find the container with id 53dba0e1b68733772438c122094e6fc876efa56623b28c26a957ad003ba1885f
Sep 11 17:25:25.594977 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:25.594949    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86kc8" event={"ID":"ea300ea1-39df-4925-add3-13a8f3823b67","Type":"ContainerStarted","Data":"53dba0e1b68733772438c122094e6fc876efa56623b28c26a957ad003ba1885f"}
Sep 11 17:25:26.636493 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:26.636449    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-r7gz4"]
Sep 11 17:25:26.645728 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:26.644984    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-r7gz4"
Sep 11 17:25:26.647819 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:26.647770    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 11 17:25:26.647819 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:26.647770    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-d9jbh\""
Sep 11 17:25:26.658582 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:26.658173    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-r7gz4"]
Sep 11 17:25:26.671209 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:26.671172    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpvxz\" (UniqueName: \"kubernetes.io/projected/c2da120d-b822-45c9-9881-a9dac929a8d2-kube-api-access-lpvxz\") pod \"dns-operator-controller-manager-7b84cd7c9c-r7gz4\" (UID: \"c2da120d-b822-45c9-9881-a9dac929a8d2\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-r7gz4"
Sep 11 17:25:26.772527 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:26.772488    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpvxz\" (UniqueName: \"kubernetes.io/projected/c2da120d-b822-45c9-9881-a9dac929a8d2-kube-api-access-lpvxz\") pod \"dns-operator-controller-manager-7b84cd7c9c-r7gz4\" (UID: \"c2da120d-b822-45c9-9881-a9dac929a8d2\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-r7gz4"
Sep 11 17:25:26.781604 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:26.781580    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpvxz\" (UniqueName: \"kubernetes.io/projected/c2da120d-b822-45c9-9881-a9dac929a8d2-kube-api-access-lpvxz\") pod \"dns-operator-controller-manager-7b84cd7c9c-r7gz4\" (UID: \"c2da120d-b822-45c9-9881-a9dac929a8d2\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-r7gz4"
Sep 11 17:25:26.959975 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:26.959886    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-r7gz4"
Sep 11 17:25:27.898157 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:27.898059    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-r7gz4"]
Sep 11 17:25:27.900778 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:25:27.900743    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2da120d_b822_45c9_9881_a9dac929a8d2.slice/crio-400fbd3097b7cfe06c3fcbd898091054f7e6dc80d4797f9775b26fbc5f0a8f9d WatchSource:0}: Error finding container 400fbd3097b7cfe06c3fcbd898091054f7e6dc80d4797f9775b26fbc5f0a8f9d: Status 404 returned error can't find the container with id 400fbd3097b7cfe06c3fcbd898091054f7e6dc80d4797f9775b26fbc5f0a8f9d
Sep 11 17:25:28.610897 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:28.610856    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-r7gz4" event={"ID":"c2da120d-b822-45c9-9881-a9dac929a8d2","Type":"ContainerStarted","Data":"400fbd3097b7cfe06c3fcbd898091054f7e6dc80d4797f9775b26fbc5f0a8f9d"}
Sep 11 17:25:29.222215 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:29.222191    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-sn5c6"]
Sep 11 17:25:29.225215 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:29.225191    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-sn5c6"
Sep 11 17:25:29.233804 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:29.233784    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-pk5hw\""
Sep 11 17:25:29.242789 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:29.242764    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-sn5c6"]
Sep 11 17:25:29.295331 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:29.295300    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mldkp\" (UniqueName: \"kubernetes.io/projected/9bdbe8b0-12cc-4c64-873c-bf6fc3106b40-kube-api-access-mldkp\") pod \"authorino-operator-76b478bd78-sn5c6\" (UID: \"9bdbe8b0-12cc-4c64-873c-bf6fc3106b40\") " pod="kuadrant-system/authorino-operator-76b478bd78-sn5c6"
Sep 11 17:25:29.396652 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:29.396622    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mldkp\" (UniqueName: \"kubernetes.io/projected/9bdbe8b0-12cc-4c64-873c-bf6fc3106b40-kube-api-access-mldkp\") pod \"authorino-operator-76b478bd78-sn5c6\" (UID: \"9bdbe8b0-12cc-4c64-873c-bf6fc3106b40\") " pod="kuadrant-system/authorino-operator-76b478bd78-sn5c6"
Sep 11 17:25:29.406454 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:29.406395    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mldkp\" (UniqueName: \"kubernetes.io/projected/9bdbe8b0-12cc-4c64-873c-bf6fc3106b40-kube-api-access-mldkp\") pod \"authorino-operator-76b478bd78-sn5c6\" (UID: \"9bdbe8b0-12cc-4c64-873c-bf6fc3106b40\") " pod="kuadrant-system/authorino-operator-76b478bd78-sn5c6"
Sep 11 17:25:29.540419 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:29.540333    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-sn5c6"
Sep 11 17:25:29.618693 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:29.618236    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86kc8" event={"ID":"ea300ea1-39df-4925-add3-13a8f3823b67","Type":"ContainerStarted","Data":"f0cec297465fc076f215ce8f735772260187bab3002413f1e7fe46e5233a9082"}
Sep 11 17:25:29.618693 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:29.618349    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86kc8"
Sep 11 17:25:29.644750 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:29.644653    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86kc8" podStartSLOduration=1.638714877 podStartE2EDuration="5.644628799s" totalImagesPullingTime="4.005913922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:25:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:25:29.64276177 +0000 UTC m=+410.036679222" watchObservedRunningTime="2026-09-11 17:25:29.644628799 +0000 UTC m=+410.038546463"
Sep 11 17:25:29.827801 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:29.827723    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-sn5c6"]
Sep 11 17:25:30.076776 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:25:30.076739    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bdbe8b0_12cc_4c64_873c_bf6fc3106b40.slice/crio-0f61f184ec5375f7d936f47a5514b083ced5b2a38066ad26ed53bdb8de4abc2c WatchSource:0}: Error finding container 0f61f184ec5375f7d936f47a5514b083ced5b2a38066ad26ed53bdb8de4abc2c: Status 404 returned error can't find the container with id 0f61f184ec5375f7d936f47a5514b083ced5b2a38066ad26ed53bdb8de4abc2c
Sep 11 17:25:30.624257 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:30.624219    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-sn5c6" event={"ID":"9bdbe8b0-12cc-4c64-873c-bf6fc3106b40","Type":"ContainerStarted","Data":"0f61f184ec5375f7d936f47a5514b083ced5b2a38066ad26ed53bdb8de4abc2c"}
Sep 11 17:25:30.625925 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:30.625898    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-r7gz4" event={"ID":"c2da120d-b822-45c9-9881-a9dac929a8d2","Type":"ContainerStarted","Data":"1e2e4fb172440fd2f6df982c3cae9899194fe5667ea36dbc72bce376e0fd63f2"}
Sep 11 17:25:30.626072 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:30.626056    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-r7gz4"
Sep 11 17:25:30.672217 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:30.672167    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-r7gz4" podStartSLOduration=2.445320332 podStartE2EDuration="4.672150463s" totalImagesPullingTime="2.226830131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:25:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:25:30.670900246 +0000 UTC m=+411.064817726" watchObservedRunningTime="2026-09-11 17:25:30.672150463 +0000 UTC m=+411.066067918"
Sep 11 17:25:32.635518 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:32.635486    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-sn5c6" event={"ID":"9bdbe8b0-12cc-4c64-873c-bf6fc3106b40","Type":"ContainerStarted","Data":"1dafe623535c8c64c0535f08820079d9aa772f3026208817f4ca1f965c7805dd"}
Sep 11 17:25:32.635912 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:32.635624    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-sn5c6"
Sep 11 17:25:32.655424 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:32.655369    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-sn5c6" podStartSLOduration=1.8578991870000001 podStartE2EDuration="3.655350629s" totalImagesPullingTime="1.797451442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:25:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:25:32.654983692 +0000 UTC m=+413.048901148" watchObservedRunningTime="2026-09-11 17:25:32.655350629 +0000 UTC m=+413.049268083"
Sep 11 17:25:40.628725 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:40.628695    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86kc8"
Sep 11 17:25:41.632830 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:41.632800    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-r7gz4"
Sep 11 17:25:42.960308 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:42.960269    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86kc8"]
Sep 11 17:25:42.960791 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:42.960525    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86kc8" podUID="ea300ea1-39df-4925-add3-13a8f3823b67" containerName="manager" containerID="cri-o://f0cec297465fc076f215ce8f735772260187bab3002413f1e7fe46e5233a9082" gracePeriod=2
Sep 11 17:25:42.971121 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:42.971092    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86kc8"]
Sep 11 17:25:42.973099 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:25:42.973060    2562 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"ea300ea1-39df-4925-add3-13a8f3823b67\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-86kc8\": pods \"kuadrant-operator-controller-manager-76f8f686bf-86kc8\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86kc8"
Sep 11 17:25:43.001383 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:43.001354    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-84fth"]
Sep 11 17:25:43.006555 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:43.006518    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea300ea1-39df-4925-add3-13a8f3823b67" containerName="manager"
Sep 11 17:25:43.006555 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:43.006558    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea300ea1-39df-4925-add3-13a8f3823b67" containerName="manager"
Sep 11 17:25:43.006806 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:43.006677    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea300ea1-39df-4925-add3-13a8f3823b67" containerName="manager"
Sep 11 17:25:43.010256 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:43.010231    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-84fth"
Sep 11 17:25:43.037310 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:25:43.037276    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-86kc8\" is forbidden: User \"system:node:ip-10-0-133-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-252.ec2.internal' and this object" podUID="ea300ea1-39df-4925-add3-13a8f3823b67" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86kc8"
Sep 11 17:25:43.068950 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:43.068916    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-84fth"]
Sep 11 17:25:43.111227 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:43.111196    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/95c15f2d-bcc9-476c-b9df-adb25518fe4e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-84fth\" (UID: \"95c15f2d-bcc9-476c-b9df-adb25518fe4e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-84fth"
Sep 11 17:25:43.111388 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:43.111244    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds86b\" (UniqueName: \"kubernetes.io/projected/95c15f2d-bcc9-476c-b9df-adb25518fe4e-kube-api-access-ds86b\") pod \"kuadrant-operator-controller-manager-76f8f686bf-84fth\" (UID: \"95c15f2d-bcc9-476c-b9df-adb25518fe4e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-84fth"
Sep 11 17:25:43.200183 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:43.200154    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86kc8"
Sep 11 17:25:43.202202 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:25:43.202172    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-86kc8\" is forbidden: User \"system:node:ip-10-0-133-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-252.ec2.internal' and this object" podUID="ea300ea1-39df-4925-add3-13a8f3823b67" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86kc8"
Sep 11 17:25:43.212598 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:43.212549    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/95c15f2d-bcc9-476c-b9df-adb25518fe4e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-84fth\" (UID: \"95c15f2d-bcc9-476c-b9df-adb25518fe4e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-84fth"
Sep 11 17:25:43.212598 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:43.212589    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds86b\" (UniqueName: \"kubernetes.io/projected/95c15f2d-bcc9-476c-b9df-adb25518fe4e-kube-api-access-ds86b\") pod \"kuadrant-operator-controller-manager-76f8f686bf-84fth\" (UID: \"95c15f2d-bcc9-476c-b9df-adb25518fe4e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-84fth"
Sep 11 17:25:43.212912 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:43.212889    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/95c15f2d-bcc9-476c-b9df-adb25518fe4e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-84fth\" (UID: \"95c15f2d-bcc9-476c-b9df-adb25518fe4e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-84fth"
Sep 11 17:25:43.223485 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:43.223459    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds86b\" (UniqueName: \"kubernetes.io/projected/95c15f2d-bcc9-476c-b9df-adb25518fe4e-kube-api-access-ds86b\") pod \"kuadrant-operator-controller-manager-76f8f686bf-84fth\" (UID: \"95c15f2d-bcc9-476c-b9df-adb25518fe4e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-84fth"
Sep 11 17:25:43.312833 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:43.312799    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mggj\" (UniqueName: \"kubernetes.io/projected/ea300ea1-39df-4925-add3-13a8f3823b67-kube-api-access-7mggj\") pod \"ea300ea1-39df-4925-add3-13a8f3823b67\" (UID: \"ea300ea1-39df-4925-add3-13a8f3823b67\") "
Sep 11 17:25:43.312833 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:43.312837    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea300ea1-39df-4925-add3-13a8f3823b67-extensions-socket-volume\") pod \"ea300ea1-39df-4925-add3-13a8f3823b67\" (UID: \"ea300ea1-39df-4925-add3-13a8f3823b67\") "
Sep 11 17:25:43.313265 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:43.313241    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea300ea1-39df-4925-add3-13a8f3823b67-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ea300ea1-39df-4925-add3-13a8f3823b67" (UID: "ea300ea1-39df-4925-add3-13a8f3823b67"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:25:43.315001 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:43.314973    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea300ea1-39df-4925-add3-13a8f3823b67-kube-api-access-7mggj" (OuterVolumeSpecName: "kube-api-access-7mggj") pod "ea300ea1-39df-4925-add3-13a8f3823b67" (UID: "ea300ea1-39df-4925-add3-13a8f3823b67"). InnerVolumeSpecName "kube-api-access-7mggj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:25:43.338168 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:43.338136    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-84fth"
Sep 11 17:25:43.413509 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:43.413480    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mggj\" (UniqueName: \"kubernetes.io/projected/ea300ea1-39df-4925-add3-13a8f3823b67-kube-api-access-7mggj\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:25:43.413509 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:43.413506    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea300ea1-39df-4925-add3-13a8f3823b67-extensions-socket-volume\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:25:43.614935 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:43.614857    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-84fth"]
Sep 11 17:25:43.617157 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:25:43.617125    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95c15f2d_bcc9_476c_b9df_adb25518fe4e.slice/crio-b57af34d3e5dce7fcc8865b5e860e2049f46bde04a30b18822394af20db2ce29 WatchSource:0}: Error finding container b57af34d3e5dce7fcc8865b5e860e2049f46bde04a30b18822394af20db2ce29: Status 404 returned error can't find the container with id b57af34d3e5dce7fcc8865b5e860e2049f46bde04a30b18822394af20db2ce29
Sep 11 17:25:43.642068 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:43.642044    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-sn5c6"
Sep 11 17:25:43.643968 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:25:43.643942    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-86kc8\" is forbidden: User \"system:node:ip-10-0-133-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-252.ec2.internal' and this object" podUID="ea300ea1-39df-4925-add3-13a8f3823b67" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86kc8"
Sep 11 17:25:43.682244 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:43.682210    2562 generic.go:356] "Generic (PLEG): container finished" podID="ea300ea1-39df-4925-add3-13a8f3823b67" containerID="f0cec297465fc076f215ce8f735772260187bab3002413f1e7fe46e5233a9082" exitCode=0
Sep 11 17:25:43.682366 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:43.682337    2562 scope.go:117] "RemoveContainer" containerID="f0cec297465fc076f215ce8f735772260187bab3002413f1e7fe46e5233a9082"
Sep 11 17:25:43.682430 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:43.682335    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86kc8"
Sep 11 17:25:43.684413 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:43.684387    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-84fth" event={"ID":"95c15f2d-bcc9-476c-b9df-adb25518fe4e","Type":"ContainerStarted","Data":"46103dc2365a2abde753f99864ce2a0c3d22c880c057d9c230746f9d8e8eb4d6"}
Sep 11 17:25:43.684519 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:43.684437    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-84fth" event={"ID":"95c15f2d-bcc9-476c-b9df-adb25518fe4e","Type":"ContainerStarted","Data":"b57af34d3e5dce7fcc8865b5e860e2049f46bde04a30b18822394af20db2ce29"}
Sep 11 17:25:43.684519 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:43.684472    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-84fth"
Sep 11 17:25:43.684634 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:25:43.684515    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-86kc8\" is forbidden: User \"system:node:ip-10-0-133-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-252.ec2.internal' and this object" podUID="ea300ea1-39df-4925-add3-13a8f3823b67" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86kc8"
Sep 11 17:25:43.687026 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:25:43.686940    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-86kc8\" is forbidden: User \"system:node:ip-10-0-133-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-252.ec2.internal' and this object" podUID="ea300ea1-39df-4925-add3-13a8f3823b67" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86kc8"
Sep 11 17:25:43.690665 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:43.690645    2562 scope.go:117] "RemoveContainer" containerID="f0cec297465fc076f215ce8f735772260187bab3002413f1e7fe46e5233a9082"
Sep 11 17:25:43.690982 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:25:43.690949    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0cec297465fc076f215ce8f735772260187bab3002413f1e7fe46e5233a9082\": container with ID starting with f0cec297465fc076f215ce8f735772260187bab3002413f1e7fe46e5233a9082 not found: ID does not exist" containerID="f0cec297465fc076f215ce8f735772260187bab3002413f1e7fe46e5233a9082"
Sep 11 17:25:43.691070 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:43.691009    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0cec297465fc076f215ce8f735772260187bab3002413f1e7fe46e5233a9082"} err="failed to get container status \"f0cec297465fc076f215ce8f735772260187bab3002413f1e7fe46e5233a9082\": rpc error: code = NotFound desc = could not find container \"f0cec297465fc076f215ce8f735772260187bab3002413f1e7fe46e5233a9082\": container with ID starting with f0cec297465fc076f215ce8f735772260187bab3002413f1e7fe46e5233a9082 not found: ID does not exist"
Sep 11 17:25:43.712603 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:43.712561    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-84fth" podStartSLOduration=1.712548503 podStartE2EDuration="1.712548503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:25:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:25:43.711250034 +0000 UTC m=+424.105167487" watchObservedRunningTime="2026-09-11 17:25:43.712548503 +0000 UTC m=+424.106465958"
Sep 11 17:25:43.714139 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:25:43.714115    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-86kc8\" is forbidden: User \"system:node:ip-10-0-133-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-252.ec2.internal' and this object" podUID="ea300ea1-39df-4925-add3-13a8f3823b67" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86kc8"
Sep 11 17:25:44.127871 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:44.127835    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea300ea1-39df-4925-add3-13a8f3823b67" path="/var/lib/kubelet/pods/ea300ea1-39df-4925-add3-13a8f3823b67/volumes"
Sep 11 17:25:54.692820 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:25:54.692788    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-84fth"
Sep 11 17:26:00.222499 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:00.222464    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-84fth"]
Sep 11 17:26:00.222902 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:00.222670    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-84fth" podUID="95c15f2d-bcc9-476c-b9df-adb25518fe4e" containerName="manager" containerID="cri-o://46103dc2365a2abde753f99864ce2a0c3d22c880c057d9c230746f9d8e8eb4d6" gracePeriod=10
Sep 11 17:26:00.471846 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:00.471820    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-84fth"
Sep 11 17:26:00.550037 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:00.549933    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/95c15f2d-bcc9-476c-b9df-adb25518fe4e-extensions-socket-volume\") pod \"95c15f2d-bcc9-476c-b9df-adb25518fe4e\" (UID: \"95c15f2d-bcc9-476c-b9df-adb25518fe4e\") "
Sep 11 17:26:00.550037 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:00.549979    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ds86b\" (UniqueName: \"kubernetes.io/projected/95c15f2d-bcc9-476c-b9df-adb25518fe4e-kube-api-access-ds86b\") pod \"95c15f2d-bcc9-476c-b9df-adb25518fe4e\" (UID: \"95c15f2d-bcc9-476c-b9df-adb25518fe4e\") "
Sep 11 17:26:00.550444 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:00.550416    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c15f2d-bcc9-476c-b9df-adb25518fe4e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "95c15f2d-bcc9-476c-b9df-adb25518fe4e" (UID: "95c15f2d-bcc9-476c-b9df-adb25518fe4e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:26:00.552113 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:00.552088    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95c15f2d-bcc9-476c-b9df-adb25518fe4e-kube-api-access-ds86b" (OuterVolumeSpecName: "kube-api-access-ds86b") pod "95c15f2d-bcc9-476c-b9df-adb25518fe4e" (UID: "95c15f2d-bcc9-476c-b9df-adb25518fe4e"). InnerVolumeSpecName "kube-api-access-ds86b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:26:00.650886 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:00.650853    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/95c15f2d-bcc9-476c-b9df-adb25518fe4e-extensions-socket-volume\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:26:00.650886 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:00.650885    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds86b\" (UniqueName: \"kubernetes.io/projected/95c15f2d-bcc9-476c-b9df-adb25518fe4e-kube-api-access-ds86b\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:26:00.756641 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:00.756603    2562 generic.go:356] "Generic (PLEG): container finished" podID="95c15f2d-bcc9-476c-b9df-adb25518fe4e" containerID="46103dc2365a2abde753f99864ce2a0c3d22c880c057d9c230746f9d8e8eb4d6" exitCode=0
Sep 11 17:26:00.756796 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:00.756652    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-84fth"
Sep 11 17:26:00.756796 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:00.756689    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-84fth" event={"ID":"95c15f2d-bcc9-476c-b9df-adb25518fe4e","Type":"ContainerDied","Data":"46103dc2365a2abde753f99864ce2a0c3d22c880c057d9c230746f9d8e8eb4d6"}
Sep 11 17:26:00.756796 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:00.756764    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-84fth" event={"ID":"95c15f2d-bcc9-476c-b9df-adb25518fe4e","Type":"ContainerDied","Data":"b57af34d3e5dce7fcc8865b5e860e2049f46bde04a30b18822394af20db2ce29"}
Sep 11 17:26:00.756796 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:00.756787    2562 scope.go:117] "RemoveContainer" containerID="46103dc2365a2abde753f99864ce2a0c3d22c880c057d9c230746f9d8e8eb4d6"
Sep 11 17:26:00.765204 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:00.765181    2562 scope.go:117] "RemoveContainer" containerID="46103dc2365a2abde753f99864ce2a0c3d22c880c057d9c230746f9d8e8eb4d6"
Sep 11 17:26:00.765465 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:26:00.765443    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46103dc2365a2abde753f99864ce2a0c3d22c880c057d9c230746f9d8e8eb4d6\": container with ID starting with 46103dc2365a2abde753f99864ce2a0c3d22c880c057d9c230746f9d8e8eb4d6 not found: ID does not exist" containerID="46103dc2365a2abde753f99864ce2a0c3d22c880c057d9c230746f9d8e8eb4d6"
Sep 11 17:26:00.765521 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:00.765477    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46103dc2365a2abde753f99864ce2a0c3d22c880c057d9c230746f9d8e8eb4d6"} err="failed to get container status \"46103dc2365a2abde753f99864ce2a0c3d22c880c057d9c230746f9d8e8eb4d6\": rpc error: code = NotFound desc = could not find container \"46103dc2365a2abde753f99864ce2a0c3d22c880c057d9c230746f9d8e8eb4d6\": container with ID starting with 46103dc2365a2abde753f99864ce2a0c3d22c880c057d9c230746f9d8e8eb4d6 not found: ID does not exist"
Sep 11 17:26:00.780971 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:00.780947    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-84fth"]
Sep 11 17:26:00.785871 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:00.785850    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-84fth"]
Sep 11 17:26:02.124241 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:02.124146    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95c15f2d-bcc9-476c-b9df-adb25518fe4e" path="/var/lib/kubelet/pods/95c15f2d-bcc9-476c-b9df-adb25518fe4e/volumes"
Sep 11 17:26:16.863735 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:16.863702    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"]
Sep 11 17:26:16.864307 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:16.864008    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95c15f2d-bcc9-476c-b9df-adb25518fe4e" containerName="manager"
Sep 11 17:26:16.864307 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:16.864025    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="95c15f2d-bcc9-476c-b9df-adb25518fe4e" containerName="manager"
Sep 11 17:26:16.864307 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:16.864089    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="95c15f2d-bcc9-476c-b9df-adb25518fe4e" containerName="manager"
Sep 11 17:26:16.866169 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:16.866150    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"
Sep 11 17:26:16.869850 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:16.869818    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-wchxf\""
Sep 11 17:26:16.886625 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:16.886598    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"]
Sep 11 17:26:16.983671 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:16.983643    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59094ae7-d124-40e9-be6a-59a658cd635d-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4ktwc\" (UID: \"59094ae7-d124-40e9-be6a-59a658cd635d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"
Sep 11 17:26:16.983842 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:16.983690    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrksz\" (UniqueName: \"kubernetes.io/projected/59094ae7-d124-40e9-be6a-59a658cd635d-kube-api-access-xrksz\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4ktwc\" (UID: \"59094ae7-d124-40e9-be6a-59a658cd635d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"
Sep 11 17:26:16.983842 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:16.983711    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59094ae7-d124-40e9-be6a-59a658cd635d-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4ktwc\" (UID: \"59094ae7-d124-40e9-be6a-59a658cd635d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"
Sep 11 17:26:16.983842 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:16.983798    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59094ae7-d124-40e9-be6a-59a658cd635d-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4ktwc\" (UID: \"59094ae7-d124-40e9-be6a-59a658cd635d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"
Sep 11 17:26:16.983973 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:16.983838    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59094ae7-d124-40e9-be6a-59a658cd635d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4ktwc\" (UID: \"59094ae7-d124-40e9-be6a-59a658cd635d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"
Sep 11 17:26:16.983973 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:16.983869    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59094ae7-d124-40e9-be6a-59a658cd635d-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4ktwc\" (UID: \"59094ae7-d124-40e9-be6a-59a658cd635d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"
Sep 11 17:26:16.983973 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:16.983895    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59094ae7-d124-40e9-be6a-59a658cd635d-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4ktwc\" (UID: \"59094ae7-d124-40e9-be6a-59a658cd635d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"
Sep 11 17:26:16.983973 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:16.983934    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59094ae7-d124-40e9-be6a-59a658cd635d-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4ktwc\" (UID: \"59094ae7-d124-40e9-be6a-59a658cd635d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"
Sep 11 17:26:16.983973 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:16.983961    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59094ae7-d124-40e9-be6a-59a658cd635d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4ktwc\" (UID: \"59094ae7-d124-40e9-be6a-59a658cd635d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"
Sep 11 17:26:17.085034 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:17.084974    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59094ae7-d124-40e9-be6a-59a658cd635d-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4ktwc\" (UID: \"59094ae7-d124-40e9-be6a-59a658cd635d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"
Sep 11 17:26:17.085034 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:17.085040    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59094ae7-d124-40e9-be6a-59a658cd635d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4ktwc\" (UID: \"59094ae7-d124-40e9-be6a-59a658cd635d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"
Sep 11 17:26:17.085283 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:17.085069    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59094ae7-d124-40e9-be6a-59a658cd635d-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4ktwc\" (UID: \"59094ae7-d124-40e9-be6a-59a658cd635d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"
Sep 11 17:26:17.085283 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:17.085089    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59094ae7-d124-40e9-be6a-59a658cd635d-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4ktwc\" (UID: \"59094ae7-d124-40e9-be6a-59a658cd635d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"
Sep 11 17:26:17.085283 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:17.085124    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59094ae7-d124-40e9-be6a-59a658cd635d-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4ktwc\" (UID: \"59094ae7-d124-40e9-be6a-59a658cd635d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"
Sep 11 17:26:17.085283 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:17.085150    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59094ae7-d124-40e9-be6a-59a658cd635d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4ktwc\" (UID: \"59094ae7-d124-40e9-be6a-59a658cd635d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"
Sep 11 17:26:17.085283 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:17.085190    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59094ae7-d124-40e9-be6a-59a658cd635d-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4ktwc\" (UID: \"59094ae7-d124-40e9-be6a-59a658cd635d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"
Sep 11 17:26:17.085283 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:17.085253    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrksz\" (UniqueName: \"kubernetes.io/projected/59094ae7-d124-40e9-be6a-59a658cd635d-kube-api-access-xrksz\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4ktwc\" (UID: \"59094ae7-d124-40e9-be6a-59a658cd635d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"
Sep 11 17:26:17.085283 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:17.085281    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59094ae7-d124-40e9-be6a-59a658cd635d-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4ktwc\" (UID: \"59094ae7-d124-40e9-be6a-59a658cd635d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"
Sep 11 17:26:17.085619 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:17.085549    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59094ae7-d124-40e9-be6a-59a658cd635d-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4ktwc\" (UID: \"59094ae7-d124-40e9-be6a-59a658cd635d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"
Sep 11 17:26:17.085619 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:17.085561    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59094ae7-d124-40e9-be6a-59a658cd635d-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4ktwc\" (UID: \"59094ae7-d124-40e9-be6a-59a658cd635d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"
Sep 11 17:26:17.085826 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:17.085775    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59094ae7-d124-40e9-be6a-59a658cd635d-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4ktwc\" (UID: \"59094ae7-d124-40e9-be6a-59a658cd635d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"
Sep 11 17:26:17.085922 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:17.085838    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59094ae7-d124-40e9-be6a-59a658cd635d-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4ktwc\" (UID: \"59094ae7-d124-40e9-be6a-59a658cd635d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"
Sep 11 17:26:17.086157 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:17.086138    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59094ae7-d124-40e9-be6a-59a658cd635d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4ktwc\" (UID: \"59094ae7-d124-40e9-be6a-59a658cd635d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"
Sep 11 17:26:17.087567 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:17.087543    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59094ae7-d124-40e9-be6a-59a658cd635d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4ktwc\" (UID: \"59094ae7-d124-40e9-be6a-59a658cd635d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"
Sep 11 17:26:17.087710 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:17.087691    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59094ae7-d124-40e9-be6a-59a658cd635d-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4ktwc\" (UID: \"59094ae7-d124-40e9-be6a-59a658cd635d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"
Sep 11 17:26:17.102311 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:17.102288    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59094ae7-d124-40e9-be6a-59a658cd635d-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4ktwc\" (UID: \"59094ae7-d124-40e9-be6a-59a658cd635d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"
Sep 11 17:26:17.107938 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:17.107915    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrksz\" (UniqueName: \"kubernetes.io/projected/59094ae7-d124-40e9-be6a-59a658cd635d-kube-api-access-xrksz\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4ktwc\" (UID: \"59094ae7-d124-40e9-be6a-59a658cd635d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"
Sep 11 17:26:17.176500 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:17.176412    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"
Sep 11 17:26:17.454122 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:17.454040    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"]
Sep 11 17:26:17.456870 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:26:17.456831    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59094ae7_d124_40e9_be6a_59a658cd635d.slice/crio-0cdad3222cc9691e76e1aaac98b06d49d1b51bf9dc1dd309690116dc05162c14 WatchSource:0}: Error finding container 0cdad3222cc9691e76e1aaac98b06d49d1b51bf9dc1dd309690116dc05162c14: Status 404 returned error can't find the container with id 0cdad3222cc9691e76e1aaac98b06d49d1b51bf9dc1dd309690116dc05162c14
Sep 11 17:26:17.828847 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:17.828764    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc" event={"ID":"59094ae7-d124-40e9-be6a-59a658cd635d","Type":"ContainerStarted","Data":"0cdad3222cc9691e76e1aaac98b06d49d1b51bf9dc1dd309690116dc05162c14"}
Sep 11 17:26:20.258737 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:20.258701    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:26:20.259030 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:20.258780    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:26:20.259030 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:20.258809    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:26:20.844361 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:20.844327    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc" event={"ID":"59094ae7-d124-40e9-be6a-59a658cd635d","Type":"ContainerStarted","Data":"d8dfbf690466ee7d160c6c5ad5d65f1a9079cbb91d0fd975919c8d5021b9cc44"}
Sep 11 17:26:20.866335 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:20.866283    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc" podStartSLOduration=2.06778299 podStartE2EDuration="4.866264526s" totalImagesPullingTime="2.798481536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:26:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:26:20.865050307 +0000 UTC m=+461.258967763" watchObservedRunningTime="2026-09-11 17:26:20.866264526 +0000 UTC m=+461.260182042"
Sep 11 17:26:21.176674 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:21.176582    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"
Sep 11 17:26:21.181939 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:21.181911    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"
Sep 11 17:26:21.848904 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:21.848867    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"
Sep 11 17:26:21.849912 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:21.849894    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"
Sep 11 17:26:21.948430 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:21.948401    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-qmdjb"]
Sep 11 17:26:21.950531 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:21.950515    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-qmdjb"
Sep 11 17:26:21.954259 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:21.954238    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-r5glb\""
Sep 11 17:26:21.954375 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:21.954239    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 11 17:26:21.961243 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:21.961218    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-qmdjb"]
Sep 11 17:26:22.029296 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:22.029264    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcqsm\" (UniqueName: \"kubernetes.io/projected/901452ef-3b0b-4084-920a-f2538f5ecd97-kube-api-access-kcqsm\") pod \"limitador-limitador-8588749977-qmdjb\" (UID: \"901452ef-3b0b-4084-920a-f2538f5ecd97\") " pod="kuadrant-system/limitador-limitador-8588749977-qmdjb"
Sep 11 17:26:22.029469 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:22.029307    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/901452ef-3b0b-4084-920a-f2538f5ecd97-config-file\") pod \"limitador-limitador-8588749977-qmdjb\" (UID: \"901452ef-3b0b-4084-920a-f2538f5ecd97\") " pod="kuadrant-system/limitador-limitador-8588749977-qmdjb"
Sep 11 17:26:22.050885 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:22.050851    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-qmdjb"]
Sep 11 17:26:22.130166 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:22.130076    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcqsm\" (UniqueName: \"kubernetes.io/projected/901452ef-3b0b-4084-920a-f2538f5ecd97-kube-api-access-kcqsm\") pod \"limitador-limitador-8588749977-qmdjb\" (UID: \"901452ef-3b0b-4084-920a-f2538f5ecd97\") " pod="kuadrant-system/limitador-limitador-8588749977-qmdjb"
Sep 11 17:26:22.130166 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:22.130120    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/901452ef-3b0b-4084-920a-f2538f5ecd97-config-file\") pod \"limitador-limitador-8588749977-qmdjb\" (UID: \"901452ef-3b0b-4084-920a-f2538f5ecd97\") " pod="kuadrant-system/limitador-limitador-8588749977-qmdjb"
Sep 11 17:26:22.130863 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:22.130841    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/901452ef-3b0b-4084-920a-f2538f5ecd97-config-file\") pod \"limitador-limitador-8588749977-qmdjb\" (UID: \"901452ef-3b0b-4084-920a-f2538f5ecd97\") " pod="kuadrant-system/limitador-limitador-8588749977-qmdjb"
Sep 11 17:26:22.142357 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:22.142312    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcqsm\" (UniqueName: \"kubernetes.io/projected/901452ef-3b0b-4084-920a-f2538f5ecd97-kube-api-access-kcqsm\") pod \"limitador-limitador-8588749977-qmdjb\" (UID: \"901452ef-3b0b-4084-920a-f2538f5ecd97\") " pod="kuadrant-system/limitador-limitador-8588749977-qmdjb"
Sep 11 17:26:22.260457 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:22.260421    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-qmdjb"
Sep 11 17:26:22.544325 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:22.544231    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-qmdjb"]
Sep 11 17:26:22.546397 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:26:22.546370    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod901452ef_3b0b_4084_920a_f2538f5ecd97.slice/crio-d8e650fcfe28f6a9bbed2d5d5d45c66faf1863afd08a1c3f48729aa88a29fb13 WatchSource:0}: Error finding container d8e650fcfe28f6a9bbed2d5d5d45c66faf1863afd08a1c3f48729aa88a29fb13: Status 404 returned error can't find the container with id d8e650fcfe28f6a9bbed2d5d5d45c66faf1863afd08a1c3f48729aa88a29fb13
Sep 11 17:26:22.854417 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:22.854385    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-qmdjb" event={"ID":"901452ef-3b0b-4084-920a-f2538f5ecd97","Type":"ContainerStarted","Data":"d8e650fcfe28f6a9bbed2d5d5d45c66faf1863afd08a1c3f48729aa88a29fb13"}
Sep 11 17:26:24.863147 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:24.863111    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-qmdjb" event={"ID":"901452ef-3b0b-4084-920a-f2538f5ecd97","Type":"ContainerStarted","Data":"3d03ecca9b945976dba0fa8cb144223d4ce41f39de5f199f862ca181df998c90"}
Sep 11 17:26:24.863634 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:24.863206    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-qmdjb"
Sep 11 17:26:24.882710 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:24.882663    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-qmdjb" podStartSLOduration=2.595711696 podStartE2EDuration="3.882649671s" totalImagesPullingTime="1.286937975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:26:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:26:24.880669755 +0000 UTC m=+465.274587213" watchObservedRunningTime="2026-09-11 17:26:24.882649671 +0000 UTC m=+465.276567164"
Sep 11 17:26:35.868516 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:35.868485    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-qmdjb"
Sep 11 17:26:38.264909 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:38.264870    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-qmdjb"]
Sep 11 17:26:38.265318 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:38.265085    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-qmdjb" podUID="901452ef-3b0b-4084-920a-f2538f5ecd97" containerName="limitador" containerID="cri-o://3d03ecca9b945976dba0fa8cb144223d4ce41f39de5f199f862ca181df998c90" gracePeriod=30
Sep 11 17:26:38.924101 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:38.924065    2562 generic.go:356] "Generic (PLEG): container finished" podID="901452ef-3b0b-4084-920a-f2538f5ecd97" containerID="3d03ecca9b945976dba0fa8cb144223d4ce41f39de5f199f862ca181df998c90" exitCode=0
Sep 11 17:26:38.924269 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:38.924116    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-qmdjb" event={"ID":"901452ef-3b0b-4084-920a-f2538f5ecd97","Type":"ContainerDied","Data":"3d03ecca9b945976dba0fa8cb144223d4ce41f39de5f199f862ca181df998c90"}
Sep 11 17:26:39.206191 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:39.206167    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-qmdjb"
Sep 11 17:26:39.263126 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:39.263090    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/901452ef-3b0b-4084-920a-f2538f5ecd97-config-file\") pod \"901452ef-3b0b-4084-920a-f2538f5ecd97\" (UID: \"901452ef-3b0b-4084-920a-f2538f5ecd97\") "
Sep 11 17:26:39.263319 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:39.263136    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcqsm\" (UniqueName: \"kubernetes.io/projected/901452ef-3b0b-4084-920a-f2538f5ecd97-kube-api-access-kcqsm\") pod \"901452ef-3b0b-4084-920a-f2538f5ecd97\" (UID: \"901452ef-3b0b-4084-920a-f2538f5ecd97\") "
Sep 11 17:26:39.263551 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:39.263515    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/901452ef-3b0b-4084-920a-f2538f5ecd97-config-file" (OuterVolumeSpecName: "config-file") pod "901452ef-3b0b-4084-920a-f2538f5ecd97" (UID: "901452ef-3b0b-4084-920a-f2538f5ecd97"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:26:39.265191 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:39.265168    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/901452ef-3b0b-4084-920a-f2538f5ecd97-kube-api-access-kcqsm" (OuterVolumeSpecName: "kube-api-access-kcqsm") pod "901452ef-3b0b-4084-920a-f2538f5ecd97" (UID: "901452ef-3b0b-4084-920a-f2538f5ecd97"). InnerVolumeSpecName "kube-api-access-kcqsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:26:39.364213 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:39.364180    2562 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/901452ef-3b0b-4084-920a-f2538f5ecd97-config-file\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:26:39.364213 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:39.364206    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcqsm\" (UniqueName: \"kubernetes.io/projected/901452ef-3b0b-4084-920a-f2538f5ecd97-kube-api-access-kcqsm\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:26:39.929326 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:39.929295    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-qmdjb" event={"ID":"901452ef-3b0b-4084-920a-f2538f5ecd97","Type":"ContainerDied","Data":"d8e650fcfe28f6a9bbed2d5d5d45c66faf1863afd08a1c3f48729aa88a29fb13"}
Sep 11 17:26:39.929326 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:39.929323    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-qmdjb"
Sep 11 17:26:39.929599 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:39.929355    2562 scope.go:117] "RemoveContainer" containerID="3d03ecca9b945976dba0fa8cb144223d4ce41f39de5f199f862ca181df998c90"
Sep 11 17:26:39.949653 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:39.949629    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-qmdjb"]
Sep 11 17:26:39.955643 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:39.955622    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-qmdjb"]
Sep 11 17:26:40.125253 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:40.125223    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="901452ef-3b0b-4084-920a-f2538f5ecd97" path="/var/lib/kubelet/pods/901452ef-3b0b-4084-920a-f2538f5ecd97/volumes"
Sep 11 17:26:43.733131 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:43.733099    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-bqkl9"]
Sep 11 17:26:43.733528 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:43.733512    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="901452ef-3b0b-4084-920a-f2538f5ecd97" containerName="limitador"
Sep 11 17:26:43.733574 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:43.733530    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="901452ef-3b0b-4084-920a-f2538f5ecd97" containerName="limitador"
Sep 11 17:26:43.733621 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:43.733605    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="901452ef-3b0b-4084-920a-f2538f5ecd97" containerName="limitador"
Sep 11 17:26:43.736563 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:43.736542    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bqkl9"
Sep 11 17:26:43.739086 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:43.739064    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 17:26:43.739334 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:43.739320    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 11 17:26:43.739905 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:43.739890    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:26:43.739961 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:43.739897    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-rbvbh\""
Sep 11 17:26:43.745896 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:43.745873    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-bqkl9"]
Sep 11 17:26:43.802288 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:43.802256    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcmm9\" (UniqueName: \"kubernetes.io/projected/389f217e-71c7-40d9-99f7-730d7b7534b4-kube-api-access-dcmm9\") pod \"postgres-69fb4594cf-bqkl9\" (UID: \"389f217e-71c7-40d9-99f7-730d7b7534b4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bqkl9"
Sep 11 17:26:43.802427 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:43.802375    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/389f217e-71c7-40d9-99f7-730d7b7534b4-data\") pod \"postgres-69fb4594cf-bqkl9\" (UID: \"389f217e-71c7-40d9-99f7-730d7b7534b4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bqkl9"
Sep 11 17:26:43.903368 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:43.903330    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/389f217e-71c7-40d9-99f7-730d7b7534b4-data\") pod \"postgres-69fb4594cf-bqkl9\" (UID: \"389f217e-71c7-40d9-99f7-730d7b7534b4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bqkl9"
Sep 11 17:26:43.903524 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:43.903383    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcmm9\" (UniqueName: \"kubernetes.io/projected/389f217e-71c7-40d9-99f7-730d7b7534b4-kube-api-access-dcmm9\") pod \"postgres-69fb4594cf-bqkl9\" (UID: \"389f217e-71c7-40d9-99f7-730d7b7534b4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bqkl9"
Sep 11 17:26:43.903804 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:43.903773    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/389f217e-71c7-40d9-99f7-730d7b7534b4-data\") pod \"postgres-69fb4594cf-bqkl9\" (UID: \"389f217e-71c7-40d9-99f7-730d7b7534b4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bqkl9"
Sep 11 17:26:43.917934 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:43.917901    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcmm9\" (UniqueName: \"kubernetes.io/projected/389f217e-71c7-40d9-99f7-730d7b7534b4-kube-api-access-dcmm9\") pod \"postgres-69fb4594cf-bqkl9\" (UID: \"389f217e-71c7-40d9-99f7-730d7b7534b4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bqkl9"
Sep 11 17:26:44.047448 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:44.047363    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bqkl9"
Sep 11 17:26:44.320781 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:44.320705    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-bqkl9"]
Sep 11 17:26:44.323819 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:26:44.323787    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod389f217e_71c7_40d9_99f7_730d7b7534b4.slice/crio-59b452d4659835d1df6d728519346fa1bc54e79520985afdbf3a3621d10711f7 WatchSource:0}: Error finding container 59b452d4659835d1df6d728519346fa1bc54e79520985afdbf3a3621d10711f7: Status 404 returned error can't find the container with id 59b452d4659835d1df6d728519346fa1bc54e79520985afdbf3a3621d10711f7
Sep 11 17:26:44.954785 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:44.954755    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bqkl9" event={"ID":"389f217e-71c7-40d9-99f7-730d7b7534b4","Type":"ContainerStarted","Data":"59b452d4659835d1df6d728519346fa1bc54e79520985afdbf3a3621d10711f7"}
Sep 11 17:26:49.980026 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:49.979966    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bqkl9" event={"ID":"389f217e-71c7-40d9-99f7-730d7b7534b4","Type":"ContainerStarted","Data":"fe977eb27ef443efb3874f301b887215b7a95e68446131c986ec4c969f6f8ca1"}
Sep 11 17:26:49.980547 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:49.980083    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bqkl9"
Sep 11 17:26:49.997070 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:49.997018    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bqkl9" podStartSLOduration=2.0158806 podStartE2EDuration="6.996982999s" totalImagesPullingTime="4.981102399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:26:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:26:49.996263944 +0000 UTC m=+490.390181398" watchObservedRunningTime="2026-09-11 17:26:49.996982999 +0000 UTC m=+490.390900456"
Sep 11 17:26:56.012862 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:26:56.012835    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bqkl9"
Sep 11 17:27:03.238900 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:03.238857    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579689bfb8-57czj"]
Sep 11 17:27:03.242006 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:03.241970    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579689bfb8-57czj"
Sep 11 17:27:03.244573 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:03.244507    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-thh5n\""
Sep 11 17:27:03.244573 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:03.244507    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 17:27:03.244760 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:03.244507    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 17:27:03.244760 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:03.244545    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 17:27:03.262047 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:03.261687    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579689bfb8-57czj"]
Sep 11 17:27:03.365940 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:03.365905    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g6nw\" (UniqueName: \"kubernetes.io/projected/663f79d4-2e9a-4e3c-ad52-42112412dd7a-kube-api-access-6g6nw\") pod \"maas-controller-579689bfb8-57czj\" (UID: \"663f79d4-2e9a-4e3c-ad52-42112412dd7a\") " pod="opendatahub/maas-controller-579689bfb8-57czj"
Sep 11 17:27:03.366129 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:03.365960    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/663f79d4-2e9a-4e3c-ad52-42112412dd7a-webhook-cert\") pod \"maas-controller-579689bfb8-57czj\" (UID: \"663f79d4-2e9a-4e3c-ad52-42112412dd7a\") " pod="opendatahub/maas-controller-579689bfb8-57czj"
Sep 11 17:27:03.366129 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:03.366078    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/663f79d4-2e9a-4e3c-ad52-42112412dd7a-metrics-certs\") pod \"maas-controller-579689bfb8-57czj\" (UID: \"663f79d4-2e9a-4e3c-ad52-42112412dd7a\") " pod="opendatahub/maas-controller-579689bfb8-57czj"
Sep 11 17:27:03.404224 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:03.404185    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7549975874-5snr4"]
Sep 11 17:27:03.408675 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:03.408646    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7549975874-5snr4"
Sep 11 17:27:03.424408 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:03.424377    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7549975874-5snr4"]
Sep 11 17:27:03.467089 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:03.467058    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/663f79d4-2e9a-4e3c-ad52-42112412dd7a-metrics-certs\") pod \"maas-controller-579689bfb8-57czj\" (UID: \"663f79d4-2e9a-4e3c-ad52-42112412dd7a\") " pod="opendatahub/maas-controller-579689bfb8-57czj"
Sep 11 17:27:03.467269 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:03.467139    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g6nw\" (UniqueName: \"kubernetes.io/projected/663f79d4-2e9a-4e3c-ad52-42112412dd7a-kube-api-access-6g6nw\") pod \"maas-controller-579689bfb8-57czj\" (UID: \"663f79d4-2e9a-4e3c-ad52-42112412dd7a\") " pod="opendatahub/maas-controller-579689bfb8-57czj"
Sep 11 17:27:03.467269 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:03.467188    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/663f79d4-2e9a-4e3c-ad52-42112412dd7a-webhook-cert\") pod \"maas-controller-579689bfb8-57czj\" (UID: \"663f79d4-2e9a-4e3c-ad52-42112412dd7a\") " pod="opendatahub/maas-controller-579689bfb8-57czj"
Sep 11 17:27:03.467269 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:27:03.467214    2562 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 11 17:27:03.467428 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:27:03.467285    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/663f79d4-2e9a-4e3c-ad52-42112412dd7a-metrics-certs podName:663f79d4-2e9a-4e3c-ad52-42112412dd7a nodeName:}" failed. No retries permitted until 2026-09-11 17:27:03.967262561 +0000 UTC m=+504.361179999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/663f79d4-2e9a-4e3c-ad52-42112412dd7a-metrics-certs") pod "maas-controller-579689bfb8-57czj" (UID: "663f79d4-2e9a-4e3c-ad52-42112412dd7a") : secret "maas-controller-metrics-tls" not found
Sep 11 17:27:03.467428 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:27:03.467291    2562 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 11 17:27:03.467428 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:27:03.467343    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/663f79d4-2e9a-4e3c-ad52-42112412dd7a-webhook-cert podName:663f79d4-2e9a-4e3c-ad52-42112412dd7a nodeName:}" failed. No retries permitted until 2026-09-11 17:27:03.967325667 +0000 UTC m=+504.361243109 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/663f79d4-2e9a-4e3c-ad52-42112412dd7a-webhook-cert") pod "maas-controller-579689bfb8-57czj" (UID: "663f79d4-2e9a-4e3c-ad52-42112412dd7a") : secret "maas-controller-webhook-cert" not found
Sep 11 17:27:03.490499 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:03.490433    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g6nw\" (UniqueName: \"kubernetes.io/projected/663f79d4-2e9a-4e3c-ad52-42112412dd7a-kube-api-access-6g6nw\") pod \"maas-controller-579689bfb8-57czj\" (UID: \"663f79d4-2e9a-4e3c-ad52-42112412dd7a\") " pod="opendatahub/maas-controller-579689bfb8-57czj"
Sep 11 17:27:03.567541 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:03.567504    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f08ec6ab-9dd1-412d-a85c-4a3802f2f895-metrics-certs\") pod \"maas-controller-7549975874-5snr4\" (UID: \"f08ec6ab-9dd1-412d-a85c-4a3802f2f895\") " pod="opendatahub/maas-controller-7549975874-5snr4"
Sep 11 17:27:03.567732 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:03.567562    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmzl4\" (UniqueName: \"kubernetes.io/projected/f08ec6ab-9dd1-412d-a85c-4a3802f2f895-kube-api-access-pmzl4\") pod \"maas-controller-7549975874-5snr4\" (UID: \"f08ec6ab-9dd1-412d-a85c-4a3802f2f895\") " pod="opendatahub/maas-controller-7549975874-5snr4"
Sep 11 17:27:03.567732 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:03.567637    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f08ec6ab-9dd1-412d-a85c-4a3802f2f895-webhook-cert\") pod \"maas-controller-7549975874-5snr4\" (UID: \"f08ec6ab-9dd1-412d-a85c-4a3802f2f895\") " pod="opendatahub/maas-controller-7549975874-5snr4"
Sep 11 17:27:03.668837 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:03.668801    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f08ec6ab-9dd1-412d-a85c-4a3802f2f895-webhook-cert\") pod \"maas-controller-7549975874-5snr4\" (UID: \"f08ec6ab-9dd1-412d-a85c-4a3802f2f895\") " pod="opendatahub/maas-controller-7549975874-5snr4"
Sep 11 17:27:03.669046 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:03.668849    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f08ec6ab-9dd1-412d-a85c-4a3802f2f895-metrics-certs\") pod \"maas-controller-7549975874-5snr4\" (UID: \"f08ec6ab-9dd1-412d-a85c-4a3802f2f895\") " pod="opendatahub/maas-controller-7549975874-5snr4"
Sep 11 17:27:03.669046 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:03.668884    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmzl4\" (UniqueName: \"kubernetes.io/projected/f08ec6ab-9dd1-412d-a85c-4a3802f2f895-kube-api-access-pmzl4\") pod \"maas-controller-7549975874-5snr4\" (UID: \"f08ec6ab-9dd1-412d-a85c-4a3802f2f895\") " pod="opendatahub/maas-controller-7549975874-5snr4"
Sep 11 17:27:03.671380 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:03.671352    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f08ec6ab-9dd1-412d-a85c-4a3802f2f895-webhook-cert\") pod \"maas-controller-7549975874-5snr4\" (UID: \"f08ec6ab-9dd1-412d-a85c-4a3802f2f895\") " pod="opendatahub/maas-controller-7549975874-5snr4"
Sep 11 17:27:03.671475 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:03.671431    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f08ec6ab-9dd1-412d-a85c-4a3802f2f895-metrics-certs\") pod \"maas-controller-7549975874-5snr4\" (UID: \"f08ec6ab-9dd1-412d-a85c-4a3802f2f895\") " pod="opendatahub/maas-controller-7549975874-5snr4"
Sep 11 17:27:03.678045 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:03.678021    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmzl4\" (UniqueName: \"kubernetes.io/projected/f08ec6ab-9dd1-412d-a85c-4a3802f2f895-kube-api-access-pmzl4\") pod \"maas-controller-7549975874-5snr4\" (UID: \"f08ec6ab-9dd1-412d-a85c-4a3802f2f895\") " pod="opendatahub/maas-controller-7549975874-5snr4"
Sep 11 17:27:03.718471 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:03.718443    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7549975874-5snr4"
Sep 11 17:27:03.971823 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:03.971791    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/663f79d4-2e9a-4e3c-ad52-42112412dd7a-metrics-certs\") pod \"maas-controller-579689bfb8-57czj\" (UID: \"663f79d4-2e9a-4e3c-ad52-42112412dd7a\") " pod="opendatahub/maas-controller-579689bfb8-57czj"
Sep 11 17:27:03.972016 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:03.971889    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/663f79d4-2e9a-4e3c-ad52-42112412dd7a-webhook-cert\") pod \"maas-controller-579689bfb8-57czj\" (UID: \"663f79d4-2e9a-4e3c-ad52-42112412dd7a\") " pod="opendatahub/maas-controller-579689bfb8-57czj"
Sep 11 17:27:03.974217 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:03.974192    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/663f79d4-2e9a-4e3c-ad52-42112412dd7a-metrics-certs\") pod \"maas-controller-579689bfb8-57czj\" (UID: \"663f79d4-2e9a-4e3c-ad52-42112412dd7a\") " pod="opendatahub/maas-controller-579689bfb8-57czj"
Sep 11 17:27:03.974330 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:03.974252    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/663f79d4-2e9a-4e3c-ad52-42112412dd7a-webhook-cert\") pod \"maas-controller-579689bfb8-57czj\" (UID: \"663f79d4-2e9a-4e3c-ad52-42112412dd7a\") " pod="opendatahub/maas-controller-579689bfb8-57czj"
Sep 11 17:27:03.987673 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:03.987645    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7549975874-5snr4"]
Sep 11 17:27:03.990013 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:27:03.989981    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf08ec6ab_9dd1_412d_a85c_4a3802f2f895.slice/crio-e3eb3a02658ecf80187b8e2cfa2754c4fde5efed8089a5478872e5074afb8451 WatchSource:0}: Error finding container e3eb3a02658ecf80187b8e2cfa2754c4fde5efed8089a5478872e5074afb8451: Status 404 returned error can't find the container with id e3eb3a02658ecf80187b8e2cfa2754c4fde5efed8089a5478872e5074afb8451
Sep 11 17:27:04.037381 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:04.037306    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7549975874-5snr4" event={"ID":"f08ec6ab-9dd1-412d-a85c-4a3802f2f895","Type":"ContainerStarted","Data":"e3eb3a02658ecf80187b8e2cfa2754c4fde5efed8089a5478872e5074afb8451"}
Sep 11 17:27:04.153942 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:04.153908    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579689bfb8-57czj"
Sep 11 17:27:04.425407 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:04.425375    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579689bfb8-57czj"]
Sep 11 17:27:04.427985 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:27:04.427957    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod663f79d4_2e9a_4e3c_ad52_42112412dd7a.slice/crio-145a79cdabff9bdf1b8e13be9310a7f9ae79ae14722726beb37d2b1f233e779a WatchSource:0}: Error finding container 145a79cdabff9bdf1b8e13be9310a7f9ae79ae14722726beb37d2b1f233e779a: Status 404 returned error can't find the container with id 145a79cdabff9bdf1b8e13be9310a7f9ae79ae14722726beb37d2b1f233e779a
Sep 11 17:27:05.048841 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:05.048801    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579689bfb8-57czj" event={"ID":"663f79d4-2e9a-4e3c-ad52-42112412dd7a","Type":"ContainerStarted","Data":"145a79cdabff9bdf1b8e13be9310a7f9ae79ae14722726beb37d2b1f233e779a"}
Sep 11 17:27:07.059166 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:07.059127    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7549975874-5snr4" event={"ID":"f08ec6ab-9dd1-412d-a85c-4a3802f2f895","Type":"ContainerStarted","Data":"e489ce91f67545b56b8d66f1eaa22b388f8fc35f10b1f5bd47f92ca3f0412fac"}
Sep 11 17:27:07.059622 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:07.059238    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7549975874-5snr4"
Sep 11 17:27:07.060648 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:07.060627    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579689bfb8-57czj" event={"ID":"663f79d4-2e9a-4e3c-ad52-42112412dd7a","Type":"ContainerStarted","Data":"91caa68aaf7aa3367bde849ebfccd29fae70601c55ae13e28ce905f8a83b605b"}
Sep 11 17:27:07.060739 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:07.060668    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579689bfb8-57czj"
Sep 11 17:27:07.080476 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:07.080430    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7549975874-5snr4" podStartSLOduration=1.527620233 podStartE2EDuration="4.080418368s" totalImagesPullingTime="2.552798135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:27:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:27:07.07912726 +0000 UTC m=+507.473044727" watchObservedRunningTime="2026-09-11 17:27:07.080418368 +0000 UTC m=+507.474335822"
Sep 11 17:27:07.104610 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:07.104562    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-579689bfb8-57czj" podStartSLOduration=1.979638641 podStartE2EDuration="4.104546052s" totalImagesPullingTime="2.124907411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:27:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:27:07.103868221 +0000 UTC m=+507.497785675" watchObservedRunningTime="2026-09-11 17:27:07.104546052 +0000 UTC m=+507.498463504"
Sep 11 17:27:18.067822 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:18.067735    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579689bfb8-57czj"
Sep 11 17:27:18.067822 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:18.067802    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7549975874-5snr4"
Sep 11 17:27:18.141728 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:18.141686    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579689bfb8-57czj"]
Sep 11 17:27:18.141918 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:18.141896    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-579689bfb8-57czj" podUID="663f79d4-2e9a-4e3c-ad52-42112412dd7a" containerName="manager" containerID="cri-o://91caa68aaf7aa3367bde849ebfccd29fae70601c55ae13e28ce905f8a83b605b" gracePeriod=30
Sep 11 17:27:18.384558 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:18.384531    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579689bfb8-57czj"
Sep 11 17:27:18.503750 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:18.503710    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6g6nw\" (UniqueName: \"kubernetes.io/projected/663f79d4-2e9a-4e3c-ad52-42112412dd7a-kube-api-access-6g6nw\") pod \"663f79d4-2e9a-4e3c-ad52-42112412dd7a\" (UID: \"663f79d4-2e9a-4e3c-ad52-42112412dd7a\") "
Sep 11 17:27:18.503750 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:18.503773    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/663f79d4-2e9a-4e3c-ad52-42112412dd7a-webhook-cert\") pod \"663f79d4-2e9a-4e3c-ad52-42112412dd7a\" (UID: \"663f79d4-2e9a-4e3c-ad52-42112412dd7a\") "
Sep 11 17:27:18.504042 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:18.503800    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/663f79d4-2e9a-4e3c-ad52-42112412dd7a-metrics-certs\") pod \"663f79d4-2e9a-4e3c-ad52-42112412dd7a\" (UID: \"663f79d4-2e9a-4e3c-ad52-42112412dd7a\") "
Sep 11 17:27:18.505941 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:18.505912    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/663f79d4-2e9a-4e3c-ad52-42112412dd7a-kube-api-access-6g6nw" (OuterVolumeSpecName: "kube-api-access-6g6nw") pod "663f79d4-2e9a-4e3c-ad52-42112412dd7a" (UID: "663f79d4-2e9a-4e3c-ad52-42112412dd7a"). InnerVolumeSpecName "kube-api-access-6g6nw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:27:18.506061 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:18.505940    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/663f79d4-2e9a-4e3c-ad52-42112412dd7a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "663f79d4-2e9a-4e3c-ad52-42112412dd7a" (UID: "663f79d4-2e9a-4e3c-ad52-42112412dd7a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:27:18.506061 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:18.506014    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/663f79d4-2e9a-4e3c-ad52-42112412dd7a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "663f79d4-2e9a-4e3c-ad52-42112412dd7a" (UID: "663f79d4-2e9a-4e3c-ad52-42112412dd7a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:27:18.604548 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:18.604504    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g6nw\" (UniqueName: \"kubernetes.io/projected/663f79d4-2e9a-4e3c-ad52-42112412dd7a-kube-api-access-6g6nw\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:27:18.604548 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:18.604550    2562 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/663f79d4-2e9a-4e3c-ad52-42112412dd7a-webhook-cert\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:27:18.604807 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:18.604566    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/663f79d4-2e9a-4e3c-ad52-42112412dd7a-metrics-certs\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:27:19.117761 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:19.117723    2562 generic.go:356] "Generic (PLEG): container finished" podID="663f79d4-2e9a-4e3c-ad52-42112412dd7a" containerID="91caa68aaf7aa3367bde849ebfccd29fae70601c55ae13e28ce905f8a83b605b" exitCode=0
Sep 11 17:27:19.118254 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:19.117770    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579689bfb8-57czj"
Sep 11 17:27:19.118254 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:19.117807    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579689bfb8-57czj" event={"ID":"663f79d4-2e9a-4e3c-ad52-42112412dd7a","Type":"ContainerDied","Data":"91caa68aaf7aa3367bde849ebfccd29fae70601c55ae13e28ce905f8a83b605b"}
Sep 11 17:27:19.118254 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:19.117867    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579689bfb8-57czj" event={"ID":"663f79d4-2e9a-4e3c-ad52-42112412dd7a","Type":"ContainerDied","Data":"145a79cdabff9bdf1b8e13be9310a7f9ae79ae14722726beb37d2b1f233e779a"}
Sep 11 17:27:19.118254 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:19.117889    2562 scope.go:117] "RemoveContainer" containerID="91caa68aaf7aa3367bde849ebfccd29fae70601c55ae13e28ce905f8a83b605b"
Sep 11 17:27:19.126742 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:19.126728    2562 scope.go:117] "RemoveContainer" containerID="91caa68aaf7aa3367bde849ebfccd29fae70601c55ae13e28ce905f8a83b605b"
Sep 11 17:27:19.127041 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:27:19.127011    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91caa68aaf7aa3367bde849ebfccd29fae70601c55ae13e28ce905f8a83b605b\": container with ID starting with 91caa68aaf7aa3367bde849ebfccd29fae70601c55ae13e28ce905f8a83b605b not found: ID does not exist" containerID="91caa68aaf7aa3367bde849ebfccd29fae70601c55ae13e28ce905f8a83b605b"
Sep 11 17:27:19.127104 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:19.127052    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91caa68aaf7aa3367bde849ebfccd29fae70601c55ae13e28ce905f8a83b605b"} err="failed to get container status \"91caa68aaf7aa3367bde849ebfccd29fae70601c55ae13e28ce905f8a83b605b\": rpc error: code = NotFound desc = could not find container \"91caa68aaf7aa3367bde849ebfccd29fae70601c55ae13e28ce905f8a83b605b\": container with ID starting with 91caa68aaf7aa3367bde849ebfccd29fae70601c55ae13e28ce905f8a83b605b not found: ID does not exist"
Sep 11 17:27:19.138513 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:19.138488    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579689bfb8-57czj"]
Sep 11 17:27:19.144468 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:19.144448    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579689bfb8-57czj"]
Sep 11 17:27:20.124810 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:20.124776    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="663f79d4-2e9a-4e3c-ad52-42112412dd7a" path="/var/lib/kubelet/pods/663f79d4-2e9a-4e3c-ad52-42112412dd7a/volumes"
Sep 11 17:27:40.182105 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:40.182076    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5ddbcf49fc-j64kl"]
Sep 11 17:27:40.182491 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:40.182358    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="663f79d4-2e9a-4e3c-ad52-42112412dd7a" containerName="manager"
Sep 11 17:27:40.182491 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:40.182368    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="663f79d4-2e9a-4e3c-ad52-42112412dd7a" containerName="manager"
Sep 11 17:27:40.182491 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:40.182427    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="663f79d4-2e9a-4e3c-ad52-42112412dd7a" containerName="manager"
Sep 11 17:27:40.188232 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:40.188207    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ddbcf49fc-j64kl"
Sep 11 17:27:40.190569 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:40.190542    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 11 17:27:40.191168 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:40.191150    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 17:27:40.191512 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:40.191496    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zp7dq\""
Sep 11 17:27:40.199058 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:40.199037    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ddbcf49fc-j64kl"]
Sep 11 17:27:40.285542 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:40.285511    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/837bcd38-4699-4d31-b335-ce9da7007b57-maas-api-tls\") pod \"maas-api-5ddbcf49fc-j64kl\" (UID: \"837bcd38-4699-4d31-b335-ce9da7007b57\") " pod="odh-ai-gateway-infra/maas-api-5ddbcf49fc-j64kl"
Sep 11 17:27:40.285735 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:40.285555    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/837bcd38-4699-4d31-b335-ce9da7007b57-openshift-service-ca\") pod \"maas-api-5ddbcf49fc-j64kl\" (UID: \"837bcd38-4699-4d31-b335-ce9da7007b57\") " pod="odh-ai-gateway-infra/maas-api-5ddbcf49fc-j64kl"
Sep 11 17:27:40.285735 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:40.285648    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/837bcd38-4699-4d31-b335-ce9da7007b57-metrics-certs\") pod \"maas-api-5ddbcf49fc-j64kl\" (UID: \"837bcd38-4699-4d31-b335-ce9da7007b57\") " pod="odh-ai-gateway-infra/maas-api-5ddbcf49fc-j64kl"
Sep 11 17:27:40.285735 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:40.285710    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4tlt\" (UniqueName: \"kubernetes.io/projected/837bcd38-4699-4d31-b335-ce9da7007b57-kube-api-access-g4tlt\") pod \"maas-api-5ddbcf49fc-j64kl\" (UID: \"837bcd38-4699-4d31-b335-ce9da7007b57\") " pod="odh-ai-gateway-infra/maas-api-5ddbcf49fc-j64kl"
Sep 11 17:27:40.387157 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:40.387122    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4tlt\" (UniqueName: \"kubernetes.io/projected/837bcd38-4699-4d31-b335-ce9da7007b57-kube-api-access-g4tlt\") pod \"maas-api-5ddbcf49fc-j64kl\" (UID: \"837bcd38-4699-4d31-b335-ce9da7007b57\") " pod="odh-ai-gateway-infra/maas-api-5ddbcf49fc-j64kl"
Sep 11 17:27:40.387359 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:40.387179    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/837bcd38-4699-4d31-b335-ce9da7007b57-maas-api-tls\") pod \"maas-api-5ddbcf49fc-j64kl\" (UID: \"837bcd38-4699-4d31-b335-ce9da7007b57\") " pod="odh-ai-gateway-infra/maas-api-5ddbcf49fc-j64kl"
Sep 11 17:27:40.387359 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:40.387219    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/837bcd38-4699-4d31-b335-ce9da7007b57-openshift-service-ca\") pod \"maas-api-5ddbcf49fc-j64kl\" (UID: \"837bcd38-4699-4d31-b335-ce9da7007b57\") " pod="odh-ai-gateway-infra/maas-api-5ddbcf49fc-j64kl"
Sep 11 17:27:40.387470 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:27:40.387360    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Sep 11 17:27:40.387470 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:40.387384    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/837bcd38-4699-4d31-b335-ce9da7007b57-metrics-certs\") pod \"maas-api-5ddbcf49fc-j64kl\" (UID: \"837bcd38-4699-4d31-b335-ce9da7007b57\") " pod="odh-ai-gateway-infra/maas-api-5ddbcf49fc-j64kl"
Sep 11 17:27:40.387470 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:27:40.387446    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/837bcd38-4699-4d31-b335-ce9da7007b57-maas-api-tls podName:837bcd38-4699-4d31-b335-ce9da7007b57 nodeName:}" failed. No retries permitted until 2026-09-11 17:27:40.887424405 +0000 UTC m=+541.281341852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/837bcd38-4699-4d31-b335-ce9da7007b57-maas-api-tls") pod "maas-api-5ddbcf49fc-j64kl" (UID: "837bcd38-4699-4d31-b335-ce9da7007b57") : secret "maas-api-serving-cert" not found
Sep 11 17:27:40.387965 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:40.387943    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/837bcd38-4699-4d31-b335-ce9da7007b57-openshift-service-ca\") pod \"maas-api-5ddbcf49fc-j64kl\" (UID: \"837bcd38-4699-4d31-b335-ce9da7007b57\") " pod="odh-ai-gateway-infra/maas-api-5ddbcf49fc-j64kl"
Sep 11 17:27:40.389857 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:40.389836    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/837bcd38-4699-4d31-b335-ce9da7007b57-metrics-certs\") pod \"maas-api-5ddbcf49fc-j64kl\" (UID: \"837bcd38-4699-4d31-b335-ce9da7007b57\") " pod="odh-ai-gateway-infra/maas-api-5ddbcf49fc-j64kl"
Sep 11 17:27:40.394718 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:40.394693    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4tlt\" (UniqueName: \"kubernetes.io/projected/837bcd38-4699-4d31-b335-ce9da7007b57-kube-api-access-g4tlt\") pod \"maas-api-5ddbcf49fc-j64kl\" (UID: \"837bcd38-4699-4d31-b335-ce9da7007b57\") " pod="odh-ai-gateway-infra/maas-api-5ddbcf49fc-j64kl"
Sep 11 17:27:40.686466 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:40.686379    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-lr9nq"]
Sep 11 17:27:40.689697 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:40.689671    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-lr9nq"
Sep 11 17:27:40.692070 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:40.691879    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-zz5j6\""
Sep 11 17:27:40.692202 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:40.691879    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 11 17:27:40.704055 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:40.704033    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-lr9nq"]
Sep 11 17:27:40.792749 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:40.792713    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbh25\" (UniqueName: \"kubernetes.io/projected/5f6b61e9-0864-47bb-aeec-df01bbf50d34-kube-api-access-nbh25\") pod \"payload-processing-5b54b7fbb7-lr9nq\" (UID: \"5f6b61e9-0864-47bb-aeec-df01bbf50d34\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-lr9nq"
Sep 11 17:27:40.792749 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:40.792758    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f6b61e9-0864-47bb-aeec-df01bbf50d34-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-lr9nq\" (UID: \"5f6b61e9-0864-47bb-aeec-df01bbf50d34\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-lr9nq"
Sep 11 17:27:40.893803 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:40.893765    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f6b61e9-0864-47bb-aeec-df01bbf50d34-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-lr9nq\" (UID: \"5f6b61e9-0864-47bb-aeec-df01bbf50d34\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-lr9nq"
Sep 11 17:27:40.894021 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:40.893847    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/837bcd38-4699-4d31-b335-ce9da7007b57-maas-api-tls\") pod \"maas-api-5ddbcf49fc-j64kl\" (UID: \"837bcd38-4699-4d31-b335-ce9da7007b57\") " pod="odh-ai-gateway-infra/maas-api-5ddbcf49fc-j64kl"
Sep 11 17:27:40.894021 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:40.893924    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbh25\" (UniqueName: \"kubernetes.io/projected/5f6b61e9-0864-47bb-aeec-df01bbf50d34-kube-api-access-nbh25\") pod \"payload-processing-5b54b7fbb7-lr9nq\" (UID: \"5f6b61e9-0864-47bb-aeec-df01bbf50d34\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-lr9nq"
Sep 11 17:27:40.894637 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:40.894616    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f6b61e9-0864-47bb-aeec-df01bbf50d34-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-lr9nq\" (UID: \"5f6b61e9-0864-47bb-aeec-df01bbf50d34\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-lr9nq"
Sep 11 17:27:40.896220 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:40.896202    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/837bcd38-4699-4d31-b335-ce9da7007b57-maas-api-tls\") pod \"maas-api-5ddbcf49fc-j64kl\" (UID: \"837bcd38-4699-4d31-b335-ce9da7007b57\") " pod="odh-ai-gateway-infra/maas-api-5ddbcf49fc-j64kl"
Sep 11 17:27:40.902089 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:40.902067    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbh25\" (UniqueName: \"kubernetes.io/projected/5f6b61e9-0864-47bb-aeec-df01bbf50d34-kube-api-access-nbh25\") pod \"payload-processing-5b54b7fbb7-lr9nq\" (UID: \"5f6b61e9-0864-47bb-aeec-df01bbf50d34\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-lr9nq"
Sep 11 17:27:41.001315 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:41.001237    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-lr9nq"
Sep 11 17:27:41.099480 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:41.099439    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ddbcf49fc-j64kl"
Sep 11 17:27:41.288972 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:41.288898    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-lr9nq"]
Sep 11 17:27:41.291905 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:27:41.291877    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f6b61e9_0864_47bb_aeec_df01bbf50d34.slice/crio-a95cb1fdb51f29ae75b8d3bb8bea5df808f1f722dbad17a8e0ec76f85f5bf298 WatchSource:0}: Error finding container a95cb1fdb51f29ae75b8d3bb8bea5df808f1f722dbad17a8e0ec76f85f5bf298: Status 404 returned error can't find the container with id a95cb1fdb51f29ae75b8d3bb8bea5df808f1f722dbad17a8e0ec76f85f5bf298
Sep 11 17:27:41.418150 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:41.418102    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ddbcf49fc-j64kl"]
Sep 11 17:27:41.423547 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:27:41.423520    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod837bcd38_4699_4d31_b335_ce9da7007b57.slice/crio-5e4b523fc5820760734ae410b25bdb3ff985216177727a1d5d2fcfe7f2fdafea WatchSource:0}: Error finding container 5e4b523fc5820760734ae410b25bdb3ff985216177727a1d5d2fcfe7f2fdafea: Status 404 returned error can't find the container with id 5e4b523fc5820760734ae410b25bdb3ff985216177727a1d5d2fcfe7f2fdafea
Sep 11 17:27:42.219059 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:42.218932    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ddbcf49fc-j64kl" event={"ID":"837bcd38-4699-4d31-b335-ce9da7007b57","Type":"ContainerStarted","Data":"5e4b523fc5820760734ae410b25bdb3ff985216177727a1d5d2fcfe7f2fdafea"}
Sep 11 17:27:42.220889 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:42.220816    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-lr9nq" event={"ID":"5f6b61e9-0864-47bb-aeec-df01bbf50d34","Type":"ContainerStarted","Data":"a95cb1fdb51f29ae75b8d3bb8bea5df808f1f722dbad17a8e0ec76f85f5bf298"}
Sep 11 17:27:45.235438 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:45.235399    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ddbcf49fc-j64kl" event={"ID":"837bcd38-4699-4d31-b335-ce9da7007b57","Type":"ContainerStarted","Data":"94b15785308aedd3985f07c841b2f65db7a25be612a412a8ff0beb6c1d12e1a9"}
Sep 11 17:27:45.236041 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:45.235506    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5ddbcf49fc-j64kl"
Sep 11 17:27:45.237114 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:45.237093    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-lr9nq" event={"ID":"5f6b61e9-0864-47bb-aeec-df01bbf50d34","Type":"ContainerStarted","Data":"5c6ad1b6d326cd4ff95ad7eeb2c84bc275f4c017d2522698746ab0b7bcb01208"}
Sep 11 17:27:45.237218 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:45.237208    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-lr9nq"
Sep 11 17:27:45.252945 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:45.252888    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5ddbcf49fc-j64kl" podStartSLOduration=1.9502944549999999 podStartE2EDuration="5.25287159s" totalImagesPullingTime="3.302577135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:27:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:27:45.25141818 +0000 UTC m=+545.645335637" watchObservedRunningTime="2026-09-11 17:27:45.25287159 +0000 UTC m=+545.646789047"
Sep 11 17:27:45.268449 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:45.268389    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-lr9nq" podStartSLOduration=1.767139492 podStartE2EDuration="5.268369704s" totalImagesPullingTime="3.501230212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:27:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:27:45.266824993 +0000 UTC m=+545.660742448" watchObservedRunningTime="2026-09-11 17:27:45.268369704 +0000 UTC m=+545.662287163"
Sep 11 17:27:50.263471 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:50.263437    2562 generic.go:356] "Generic (PLEG): container finished" podID="59094ae7-d124-40e9-be6a-59a658cd635d" containerID="d8dfbf690466ee7d160c6c5ad5d65f1a9079cbb91d0fd975919c8d5021b9cc44" exitCode=0
Sep 11 17:27:50.263895 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:50.263510    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc" event={"ID":"59094ae7-d124-40e9-be6a-59a658cd635d","Type":"ContainerDied","Data":"d8dfbf690466ee7d160c6c5ad5d65f1a9079cbb91d0fd975919c8d5021b9cc44"}
Sep 11 17:27:50.263895 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:50.263887    2562 scope.go:117] "RemoveContainer" containerID="d8dfbf690466ee7d160c6c5ad5d65f1a9079cbb91d0fd975919c8d5021b9cc44"
Sep 11 17:27:50.264818 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:50.264794    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:27:50.264869 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:50.264858    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:27:50.264903 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:50.264885    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:27:51.252685 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:51.252640    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5ddbcf49fc-j64kl"
Sep 11 17:27:51.272923 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:51.272883    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc" event={"ID":"59094ae7-d124-40e9-be6a-59a658cd635d","Type":"ContainerStarted","Data":"f675b46267d466a73bc5b97b0b88bc061e98676f519ab2a8fbfb80773e29069e"}
Sep 11 17:27:51.849004 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:51.848945    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"
Sep 11 17:27:52.177550 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:52.177469    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"
Sep 11 17:27:52.178573 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:52.178543    2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-4ktwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body=
Sep 11 17:27:52.178690 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:52.178621    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc" podUID="59094ae7-d124-40e9-be6a-59a658cd635d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused"
Sep 11 17:27:53.177557 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:53.177517    2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-4ktwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body=
Sep 11 17:27:53.177955 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:53.177585    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc" podUID="59094ae7-d124-40e9-be6a-59a658cd635d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused"
Sep 11 17:27:54.177855 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:54.177812    2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-4ktwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body=
Sep 11 17:27:54.178304 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:54.177875    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc" podUID="59094ae7-d124-40e9-be6a-59a658cd635d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused"
Sep 11 17:27:55.176887 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:55.176844    2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-4ktwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body=
Sep 11 17:27:55.177118 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:55.176907    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc" podUID="59094ae7-d124-40e9-be6a-59a658cd635d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused"
Sep 11 17:27:56.177346 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:56.177308    2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-4ktwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body=
Sep 11 17:27:56.177747 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:56.177377    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc" podUID="59094ae7-d124-40e9-be6a-59a658cd635d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused"
Sep 11 17:27:56.245225 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:56.245190    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-lr9nq"
Sep 11 17:27:57.180536 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:57.180507    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"
Sep 11 17:27:57.181666 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:27:57.181639    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4ktwc"
Sep 11 17:28:16.532947 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:16.532914    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-gndz7"]
Sep 11 17:28:16.584878 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:16.584840    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-gndz7"]
Sep 11 17:28:16.585061 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:16.584948    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gndz7"
Sep 11 17:28:16.590056 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:16.590028    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 17:28:16.590056 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:16.590040    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 17:28:16.590260 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:16.590035    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-d66tb\""
Sep 11 17:28:16.590260 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:16.590038    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 11 17:28:16.613329 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:16.613299    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cecd860f-bb94-43b0-93f8-967f4a227add-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gndz7\" (UID: \"cecd860f-bb94-43b0-93f8-967f4a227add\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gndz7"
Sep 11 17:28:16.613329 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:16.613329    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pckw6\" (UniqueName: \"kubernetes.io/projected/cecd860f-bb94-43b0-93f8-967f4a227add-kube-api-access-pckw6\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gndz7\" (UID: \"cecd860f-bb94-43b0-93f8-967f4a227add\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gndz7"
Sep 11 17:28:16.613556 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:16.613352    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cecd860f-bb94-43b0-93f8-967f4a227add-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gndz7\" (UID: \"cecd860f-bb94-43b0-93f8-967f4a227add\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gndz7"
Sep 11 17:28:16.613556 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:16.613455    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cecd860f-bb94-43b0-93f8-967f4a227add-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gndz7\" (UID: \"cecd860f-bb94-43b0-93f8-967f4a227add\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gndz7"
Sep 11 17:28:16.613556 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:16.613504    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cecd860f-bb94-43b0-93f8-967f4a227add-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gndz7\" (UID: \"cecd860f-bb94-43b0-93f8-967f4a227add\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gndz7"
Sep 11 17:28:16.613556 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:16.613551    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cecd860f-bb94-43b0-93f8-967f4a227add-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gndz7\" (UID: \"cecd860f-bb94-43b0-93f8-967f4a227add\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gndz7"
Sep 11 17:28:16.714873 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:16.714834    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cecd860f-bb94-43b0-93f8-967f4a227add-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gndz7\" (UID: \"cecd860f-bb94-43b0-93f8-967f4a227add\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gndz7"
Sep 11 17:28:16.714873 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:16.714872    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pckw6\" (UniqueName: \"kubernetes.io/projected/cecd860f-bb94-43b0-93f8-967f4a227add-kube-api-access-pckw6\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gndz7\" (UID: \"cecd860f-bb94-43b0-93f8-967f4a227add\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gndz7"
Sep 11 17:28:16.715152 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:16.714910    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cecd860f-bb94-43b0-93f8-967f4a227add-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gndz7\" (UID: \"cecd860f-bb94-43b0-93f8-967f4a227add\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gndz7"
Sep 11 17:28:16.715152 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:16.714948    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cecd860f-bb94-43b0-93f8-967f4a227add-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gndz7\" (UID: \"cecd860f-bb94-43b0-93f8-967f4a227add\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gndz7"
Sep 11 17:28:16.715152 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:16.714987    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cecd860f-bb94-43b0-93f8-967f4a227add-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gndz7\" (UID: \"cecd860f-bb94-43b0-93f8-967f4a227add\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gndz7"
Sep 11 17:28:16.715152 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:16.715035    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cecd860f-bb94-43b0-93f8-967f4a227add-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gndz7\" (UID: \"cecd860f-bb94-43b0-93f8-967f4a227add\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gndz7"
Sep 11 17:28:16.715360 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:16.715340    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cecd860f-bb94-43b0-93f8-967f4a227add-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gndz7\" (UID: \"cecd860f-bb94-43b0-93f8-967f4a227add\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gndz7"
Sep 11 17:28:16.715456 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:16.715434    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cecd860f-bb94-43b0-93f8-967f4a227add-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gndz7\" (UID: \"cecd860f-bb94-43b0-93f8-967f4a227add\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gndz7"
Sep 11 17:28:16.715456 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:16.715446    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cecd860f-bb94-43b0-93f8-967f4a227add-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gndz7\" (UID: \"cecd860f-bb94-43b0-93f8-967f4a227add\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gndz7"
Sep 11 17:28:16.717160 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:16.717138    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cecd860f-bb94-43b0-93f8-967f4a227add-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gndz7\" (UID: \"cecd860f-bb94-43b0-93f8-967f4a227add\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gndz7"
Sep 11 17:28:16.717548 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:16.717531    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cecd860f-bb94-43b0-93f8-967f4a227add-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gndz7\" (UID: \"cecd860f-bb94-43b0-93f8-967f4a227add\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gndz7"
Sep 11 17:28:16.725184 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:16.725163    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pckw6\" (UniqueName: \"kubernetes.io/projected/cecd860f-bb94-43b0-93f8-967f4a227add-kube-api-access-pckw6\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gndz7\" (UID: \"cecd860f-bb94-43b0-93f8-967f4a227add\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gndz7"
Sep 11 17:28:16.895558 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:16.895519    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gndz7"
Sep 11 17:28:17.164737 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:17.164704    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-gndz7"]
Sep 11 17:28:17.166836 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:28:17.166805    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcecd860f_bb94_43b0_93f8_967f4a227add.slice/crio-8a3d54021831ebb697e318e09722a3a0288af559afd7a5df9eda61052d00b490 WatchSource:0}: Error finding container 8a3d54021831ebb697e318e09722a3a0288af559afd7a5df9eda61052d00b490: Status 404 returned error can't find the container with id 8a3d54021831ebb697e318e09722a3a0288af559afd7a5df9eda61052d00b490
Sep 11 17:28:17.387332 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:17.387289    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gndz7" event={"ID":"cecd860f-bb94-43b0-93f8-967f4a227add","Type":"ContainerStarted","Data":"8a3d54021831ebb697e318e09722a3a0288af559afd7a5df9eda61052d00b490"}
Sep 11 17:28:18.392907 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:18.392868    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gndz7" event={"ID":"cecd860f-bb94-43b0-93f8-967f4a227add","Type":"ContainerStarted","Data":"eb386fa0dea336ce25626c1f60885386c898a95bc167cf89bba4b7a6a2ede348"}
Sep 11 17:28:18.412267 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:18.412165    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gndz7" podStartSLOduration=1.429449693 podStartE2EDuration="2.41214833s" totalImagesPullingTime="982.698637ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:28:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:28:18.410277653 +0000 UTC m=+578.804195108" watchObservedRunningTime="2026-09-11 17:28:18.41214833 +0000 UTC m=+578.806065784"
Sep 11 17:28:26.895876 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:26.895835    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gndz7"
Sep 11 17:28:26.896486 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:26.895917    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gndz7"
Sep 11 17:28:26.908528 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:26.908504    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gndz7"
Sep 11 17:28:27.442256 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:27.442229    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gndz7"
Sep 11 17:28:43.795599 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:43.795553    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f"]
Sep 11 17:28:43.799399 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:43.799377    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f"
Sep 11 17:28:43.801625 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:43.801601    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 11 17:28:43.808391 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:43.808371    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f"]
Sep 11 17:28:43.957056 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:43.957018    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7s7t\" (UniqueName: \"kubernetes.io/projected/31de6239-94e0-410b-97fd-a613b33b12c2-kube-api-access-t7s7t\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f\" (UID: \"31de6239-94e0-410b-97fd-a613b33b12c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f"
Sep 11 17:28:43.957215 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:43.957091    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31de6239-94e0-410b-97fd-a613b33b12c2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f\" (UID: \"31de6239-94e0-410b-97fd-a613b33b12c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f"
Sep 11 17:28:43.957215 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:43.957113    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31de6239-94e0-410b-97fd-a613b33b12c2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f\" (UID: \"31de6239-94e0-410b-97fd-a613b33b12c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f"
Sep 11 17:28:43.957215 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:43.957131    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31de6239-94e0-410b-97fd-a613b33b12c2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f\" (UID: \"31de6239-94e0-410b-97fd-a613b33b12c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f"
Sep 11 17:28:43.957215 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:43.957168    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31de6239-94e0-410b-97fd-a613b33b12c2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f\" (UID: \"31de6239-94e0-410b-97fd-a613b33b12c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f"
Sep 11 17:28:43.957348 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:43.957217    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31de6239-94e0-410b-97fd-a613b33b12c2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f\" (UID: \"31de6239-94e0-410b-97fd-a613b33b12c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f"
Sep 11 17:28:44.058397 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:44.058302    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31de6239-94e0-410b-97fd-a613b33b12c2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f\" (UID: \"31de6239-94e0-410b-97fd-a613b33b12c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f"
Sep 11 17:28:44.058397 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:44.058343    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31de6239-94e0-410b-97fd-a613b33b12c2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f\" (UID: \"31de6239-94e0-410b-97fd-a613b33b12c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f"
Sep 11 17:28:44.058397 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:44.058363    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31de6239-94e0-410b-97fd-a613b33b12c2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f\" (UID: \"31de6239-94e0-410b-97fd-a613b33b12c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f"
Sep 11 17:28:44.058397 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:44.058388    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31de6239-94e0-410b-97fd-a613b33b12c2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f\" (UID: \"31de6239-94e0-410b-97fd-a613b33b12c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f"
Sep 11 17:28:44.058750 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:44.058416    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31de6239-94e0-410b-97fd-a613b33b12c2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f\" (UID: \"31de6239-94e0-410b-97fd-a613b33b12c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f"
Sep 11 17:28:44.058750 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:44.058465    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7s7t\" (UniqueName: \"kubernetes.io/projected/31de6239-94e0-410b-97fd-a613b33b12c2-kube-api-access-t7s7t\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f\" (UID: \"31de6239-94e0-410b-97fd-a613b33b12c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f"
Sep 11 17:28:44.058864 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:44.058808    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31de6239-94e0-410b-97fd-a613b33b12c2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f\" (UID: \"31de6239-94e0-410b-97fd-a613b33b12c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f"
Sep 11 17:28:44.058909 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:44.058880    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31de6239-94e0-410b-97fd-a613b33b12c2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f\" (UID: \"31de6239-94e0-410b-97fd-a613b33b12c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f"
Sep 11 17:28:44.058909 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:44.058891    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31de6239-94e0-410b-97fd-a613b33b12c2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f\" (UID: \"31de6239-94e0-410b-97fd-a613b33b12c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f"
Sep 11 17:28:44.060697 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:44.060671    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31de6239-94e0-410b-97fd-a613b33b12c2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f\" (UID: \"31de6239-94e0-410b-97fd-a613b33b12c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f"
Sep 11 17:28:44.060938 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:44.060923    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31de6239-94e0-410b-97fd-a613b33b12c2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f\" (UID: \"31de6239-94e0-410b-97fd-a613b33b12c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f"
Sep 11 17:28:44.066394 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:44.066372    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7s7t\" (UniqueName: \"kubernetes.io/projected/31de6239-94e0-410b-97fd-a613b33b12c2-kube-api-access-t7s7t\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f\" (UID: \"31de6239-94e0-410b-97fd-a613b33b12c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f"
Sep 11 17:28:44.109598 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:44.109565    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f"
Sep 11 17:28:44.381031 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:44.380985    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f"]
Sep 11 17:28:44.383347 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:28:44.383315    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31de6239_94e0_410b_97fd_a613b33b12c2.slice/crio-2a8dc58fa1cdb157f30d32a05aa93cc759e6e81d90573dd23f4407f3c38f1546 WatchSource:0}: Error finding container 2a8dc58fa1cdb157f30d32a05aa93cc759e6e81d90573dd23f4407f3c38f1546: Status 404 returned error can't find the container with id 2a8dc58fa1cdb157f30d32a05aa93cc759e6e81d90573dd23f4407f3c38f1546
Sep 11 17:28:44.508064 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:44.508031    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f" event={"ID":"31de6239-94e0-410b-97fd-a613b33b12c2","Type":"ContainerStarted","Data":"2a8dc58fa1cdb157f30d32a05aa93cc759e6e81d90573dd23f4407f3c38f1546"}
Sep 11 17:28:45.513976 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:45.513937    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f" event={"ID":"31de6239-94e0-410b-97fd-a613b33b12c2","Type":"ContainerStarted","Data":"ec4fae8a7b73747d8395c0b689f4d5f444cef50a943610bb9e5d0756b20d5714"}
Sep 11 17:28:45.531537 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:45.531488    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f" podStartSLOduration=2.333002532 podStartE2EDuration="2.531470256s" totalImagesPullingTime="198.467724ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:28:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:28:45.530280074 +0000 UTC m=+605.924197528" watchObservedRunningTime="2026-09-11 17:28:45.531470256 +0000 UTC m=+605.925387768"
Sep 11 17:28:54.109805 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:54.109766    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f"
Sep 11 17:28:54.110473 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:54.109842    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f"
Sep 11 17:28:54.125825 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:54.125792    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f"
Sep 11 17:28:54.566216 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:28:54.566118    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f"
Sep 11 17:31:09.565708 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:09.565673    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"]
Sep 11 17:31:09.569538 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:09.569513    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"
Sep 11 17:31:09.571904 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:09.571880    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-c7d192-openshift-default-dockercfg-jcqrd\""
Sep 11 17:31:09.584492 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:09.584468    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"]
Sep 11 17:31:09.759236 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:09.759185    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00b26df4-2ff6-45f3-b020-e1854b2060d6-istio-envoy\") pod \"e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4\" (UID: \"00b26df4-2ff6-45f3-b020-e1854b2060d6\") " pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"
Sep 11 17:31:09.759436 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:09.759262    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00b26df4-2ff6-45f3-b020-e1854b2060d6-istiod-ca-cert\") pod \"e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4\" (UID: \"00b26df4-2ff6-45f3-b020-e1854b2060d6\") " pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"
Sep 11 17:31:09.759436 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:09.759304    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b26df4-2ff6-45f3-b020-e1854b2060d6-workload-socket\") pod \"e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4\" (UID: \"00b26df4-2ff6-45f3-b020-e1854b2060d6\") " pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"
Sep 11 17:31:09.759436 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:09.759334    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8mzp\" (UniqueName: \"kubernetes.io/projected/00b26df4-2ff6-45f3-b020-e1854b2060d6-kube-api-access-l8mzp\") pod \"e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4\" (UID: \"00b26df4-2ff6-45f3-b020-e1854b2060d6\") " pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"
Sep 11 17:31:09.759605 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:09.759437    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00b26df4-2ff6-45f3-b020-e1854b2060d6-istio-data\") pod \"e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4\" (UID: \"00b26df4-2ff6-45f3-b020-e1854b2060d6\") " pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"
Sep 11 17:31:09.759605 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:09.759475    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b26df4-2ff6-45f3-b020-e1854b2060d6-credential-socket\") pod \"e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4\" (UID: \"00b26df4-2ff6-45f3-b020-e1854b2060d6\") " pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"
Sep 11 17:31:09.759605 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:09.759536    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00b26df4-2ff6-45f3-b020-e1854b2060d6-istio-token\") pod \"e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4\" (UID: \"00b26df4-2ff6-45f3-b020-e1854b2060d6\") " pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"
Sep 11 17:31:09.759605 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:09.759579    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00b26df4-2ff6-45f3-b020-e1854b2060d6-workload-certs\") pod \"e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4\" (UID: \"00b26df4-2ff6-45f3-b020-e1854b2060d6\") " pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"
Sep 11 17:31:09.759754 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:09.759607    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00b26df4-2ff6-45f3-b020-e1854b2060d6-istio-podinfo\") pod \"e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4\" (UID: \"00b26df4-2ff6-45f3-b020-e1854b2060d6\") " pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"
Sep 11 17:31:09.860326 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:09.860285    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00b26df4-2ff6-45f3-b020-e1854b2060d6-istio-envoy\") pod \"e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4\" (UID: \"00b26df4-2ff6-45f3-b020-e1854b2060d6\") " pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"
Sep 11 17:31:09.860501 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:09.860338    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00b26df4-2ff6-45f3-b020-e1854b2060d6-istiod-ca-cert\") pod \"e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4\" (UID: \"00b26df4-2ff6-45f3-b020-e1854b2060d6\") " pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"
Sep 11 17:31:09.860501 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:09.860457    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b26df4-2ff6-45f3-b020-e1854b2060d6-workload-socket\") pod \"e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4\" (UID: \"00b26df4-2ff6-45f3-b020-e1854b2060d6\") " pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"
Sep 11 17:31:09.860501 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:09.860492    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8mzp\" (UniqueName: \"kubernetes.io/projected/00b26df4-2ff6-45f3-b020-e1854b2060d6-kube-api-access-l8mzp\") pod \"e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4\" (UID: \"00b26df4-2ff6-45f3-b020-e1854b2060d6\") " pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"
Sep 11 17:31:09.860622 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:09.860519    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00b26df4-2ff6-45f3-b020-e1854b2060d6-istio-data\") pod \"e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4\" (UID: \"00b26df4-2ff6-45f3-b020-e1854b2060d6\") " pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"
Sep 11 17:31:09.860622 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:09.860560    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b26df4-2ff6-45f3-b020-e1854b2060d6-credential-socket\") pod \"e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4\" (UID: \"00b26df4-2ff6-45f3-b020-e1854b2060d6\") " pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"
Sep 11 17:31:09.860622 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:09.860587    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00b26df4-2ff6-45f3-b020-e1854b2060d6-istio-token\") pod \"e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4\" (UID: \"00b26df4-2ff6-45f3-b020-e1854b2060d6\") " pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"
Sep 11 17:31:09.860622 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:09.860615    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00b26df4-2ff6-45f3-b020-e1854b2060d6-workload-certs\") pod \"e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4\" (UID: \"00b26df4-2ff6-45f3-b020-e1854b2060d6\") " pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"
Sep 11 17:31:09.860809 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:09.860639    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00b26df4-2ff6-45f3-b020-e1854b2060d6-istio-podinfo\") pod \"e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4\" (UID: \"00b26df4-2ff6-45f3-b020-e1854b2060d6\") " pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"
Sep 11 17:31:09.860920 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:09.860894    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00b26df4-2ff6-45f3-b020-e1854b2060d6-istio-data\") pod \"e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4\" (UID: \"00b26df4-2ff6-45f3-b020-e1854b2060d6\") " pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"
Sep 11 17:31:09.860986 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:09.860938    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b26df4-2ff6-45f3-b020-e1854b2060d6-credential-socket\") pod \"e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4\" (UID: \"00b26df4-2ff6-45f3-b020-e1854b2060d6\") " pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"
Sep 11 17:31:09.861091 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:09.861070    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b26df4-2ff6-45f3-b020-e1854b2060d6-workload-socket\") pod \"e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4\" (UID: \"00b26df4-2ff6-45f3-b020-e1854b2060d6\") " pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"
Sep 11 17:31:09.861185 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:09.861164    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00b26df4-2ff6-45f3-b020-e1854b2060d6-workload-certs\") pod \"e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4\" (UID: \"00b26df4-2ff6-45f3-b020-e1854b2060d6\") " pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"
Sep 11 17:31:09.861236 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:09.861197    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00b26df4-2ff6-45f3-b020-e1854b2060d6-istiod-ca-cert\") pod \"e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4\" (UID: \"00b26df4-2ff6-45f3-b020-e1854b2060d6\") " pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"
Sep 11 17:31:09.862665 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:09.862643    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00b26df4-2ff6-45f3-b020-e1854b2060d6-istio-envoy\") pod \"e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4\" (UID: \"00b26df4-2ff6-45f3-b020-e1854b2060d6\") " pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"
Sep 11 17:31:09.863125 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:09.863102    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00b26df4-2ff6-45f3-b020-e1854b2060d6-istio-podinfo\") pod \"e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4\" (UID: \"00b26df4-2ff6-45f3-b020-e1854b2060d6\") " pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"
Sep 11 17:31:09.870136 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:09.870111    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00b26df4-2ff6-45f3-b020-e1854b2060d6-istio-token\") pod \"e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4\" (UID: \"00b26df4-2ff6-45f3-b020-e1854b2060d6\") " pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"
Sep 11 17:31:09.870235 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:09.870143    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8mzp\" (UniqueName: \"kubernetes.io/projected/00b26df4-2ff6-45f3-b020-e1854b2060d6-kube-api-access-l8mzp\") pod \"e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4\" (UID: \"00b26df4-2ff6-45f3-b020-e1854b2060d6\") " pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"
Sep 11 17:31:09.881934 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:09.881910    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"
Sep 11 17:31:10.158655 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:10.158629    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"]
Sep 11 17:31:10.160923 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:31:10.160895    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00b26df4_2ff6_45f3_b020_e1854b2060d6.slice/crio-d1518d53fb8dbf3f272fd7976ef6657be64e269a18e76ff322316c4fccaf4d0c WatchSource:0}: Error finding container d1518d53fb8dbf3f272fd7976ef6657be64e269a18e76ff322316c4fccaf4d0c: Status 404 returned error can't find the container with id d1518d53fb8dbf3f272fd7976ef6657be64e269a18e76ff322316c4fccaf4d0c
Sep 11 17:31:10.163646 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:10.163607    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:31:10.163768 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:10.163684    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:31:10.163768 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:10.163718    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:31:11.136610 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:11.136570    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4" event={"ID":"00b26df4-2ff6-45f3-b020-e1854b2060d6","Type":"ContainerStarted","Data":"22f018c6731d122966eae74773279a0812791889fc4c6b848a8f54a8a32d1332"}
Sep 11 17:31:11.137048 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:11.136633    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4" event={"ID":"00b26df4-2ff6-45f3-b020-e1854b2060d6","Type":"ContainerStarted","Data":"d1518d53fb8dbf3f272fd7976ef6657be64e269a18e76ff322316c4fccaf4d0c"}
Sep 11 17:31:11.163130 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:11.163084    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4" podStartSLOduration=2.163070041 podStartE2EDuration="2.163070041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:31:11.16141389 +0000 UTC m=+751.555331346" watchObservedRunningTime="2026-09-11 17:31:11.163070041 +0000 UTC m=+751.556987496"
Sep 11 17:31:11.882479 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:11.882441    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"
Sep 11 17:31:11.887507 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:11.887482    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"
Sep 11 17:31:12.141007 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:12.140893    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"
Sep 11 17:31:12.141811 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:12.141779    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"
Sep 11 17:31:14.067369 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:14.067303    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq"]
Sep 11 17:31:14.071389 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:14.071353    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq"
Sep 11 17:31:14.073878 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:14.073856    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-144bfa-serving-cert\""
Sep 11 17:31:14.084611 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:14.084580    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq"]
Sep 11 17:31:14.101082 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:14.101054    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd26d755-6e39-49b6-b956-98d5709b58b0-metrics-certs\") pod \"maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq\" (UID: \"dd26d755-6e39-49b6-b956-98d5709b58b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq"
Sep 11 17:31:14.101195 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:14.101168    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd26d755-6e39-49b6-b956-98d5709b58b0-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq\" (UID: \"dd26d755-6e39-49b6-b956-98d5709b58b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq"
Sep 11 17:31:14.101239 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:14.101220    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd26d755-6e39-49b6-b956-98d5709b58b0-maas-api-tls\") pod \"maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq\" (UID: \"dd26d755-6e39-49b6-b956-98d5709b58b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq"
Sep 11 17:31:14.101287 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:14.101270    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62kfm\" (UniqueName: \"kubernetes.io/projected/dd26d755-6e39-49b6-b956-98d5709b58b0-kube-api-access-62kfm\") pod \"maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq\" (UID: \"dd26d755-6e39-49b6-b956-98d5709b58b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq"
Sep 11 17:31:14.201987 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:14.201950    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd26d755-6e39-49b6-b956-98d5709b58b0-metrics-certs\") pod \"maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq\" (UID: \"dd26d755-6e39-49b6-b956-98d5709b58b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq"
Sep 11 17:31:14.202203 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:14.202079    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd26d755-6e39-49b6-b956-98d5709b58b0-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq\" (UID: \"dd26d755-6e39-49b6-b956-98d5709b58b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq"
Sep 11 17:31:14.202203 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:14.202111    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd26d755-6e39-49b6-b956-98d5709b58b0-maas-api-tls\") pod \"maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq\" (UID: \"dd26d755-6e39-49b6-b956-98d5709b58b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq"
Sep 11 17:31:14.202203 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:14.202157    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62kfm\" (UniqueName: \"kubernetes.io/projected/dd26d755-6e39-49b6-b956-98d5709b58b0-kube-api-access-62kfm\") pod \"maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq\" (UID: \"dd26d755-6e39-49b6-b956-98d5709b58b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq"
Sep 11 17:31:14.202417 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:31:14.202393    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-144bfa-serving-cert: secret "maas-api-e2e-worker-w4-144bfa-serving-cert" not found
Sep 11 17:31:14.202519 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:31:14.202501    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd26d755-6e39-49b6-b956-98d5709b58b0-maas-api-tls podName:dd26d755-6e39-49b6-b956-98d5709b58b0 nodeName:}" failed. No retries permitted until 2026-09-11 17:31:14.702478013 +0000 UTC m=+755.096395446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dd26d755-6e39-49b6-b956-98d5709b58b0-maas-api-tls") pod "maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq" (UID: "dd26d755-6e39-49b6-b956-98d5709b58b0") : secret "maas-api-e2e-worker-w4-144bfa-serving-cert" not found
Sep 11 17:31:14.202833 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:14.202809    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd26d755-6e39-49b6-b956-98d5709b58b0-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq\" (UID: \"dd26d755-6e39-49b6-b956-98d5709b58b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq"
Sep 11 17:31:14.204921 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:14.204902    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd26d755-6e39-49b6-b956-98d5709b58b0-metrics-certs\") pod \"maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq\" (UID: \"dd26d755-6e39-49b6-b956-98d5709b58b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq"
Sep 11 17:31:14.216440 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:14.216410    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62kfm\" (UniqueName: \"kubernetes.io/projected/dd26d755-6e39-49b6-b956-98d5709b58b0-kube-api-access-62kfm\") pod \"maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq\" (UID: \"dd26d755-6e39-49b6-b956-98d5709b58b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq"
Sep 11 17:31:14.706403 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:14.706364    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd26d755-6e39-49b6-b956-98d5709b58b0-maas-api-tls\") pod \"maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq\" (UID: \"dd26d755-6e39-49b6-b956-98d5709b58b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq"
Sep 11 17:31:14.708915 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:14.708883    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd26d755-6e39-49b6-b956-98d5709b58b0-maas-api-tls\") pod \"maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq\" (UID: \"dd26d755-6e39-49b6-b956-98d5709b58b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq"
Sep 11 17:31:14.987647 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:14.987559    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq"
Sep 11 17:31:15.196785 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:15.196754    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w"]
Sep 11 17:31:15.202114 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:15.202088    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w"
Sep 11 17:31:15.205574 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:15.205552    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-08c6b2-serving-cert\""
Sep 11 17:31:15.211698 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:15.211631    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkw4l\" (UniqueName: \"kubernetes.io/projected/5f249ed3-acb4-4e5d-b189-de21af9f1a25-kube-api-access-tkw4l\") pod \"maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w\" (UID: \"5f249ed3-acb4-4e5d-b189-de21af9f1a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w"
Sep 11 17:31:15.211698 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:15.211679    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f249ed3-acb4-4e5d-b189-de21af9f1a25-metrics-certs\") pod \"maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w\" (UID: \"5f249ed3-acb4-4e5d-b189-de21af9f1a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w"
Sep 11 17:31:15.211965 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:15.211738    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f249ed3-acb4-4e5d-b189-de21af9f1a25-maas-api-tls\") pod \"maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w\" (UID: \"5f249ed3-acb4-4e5d-b189-de21af9f1a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w"
Sep 11 17:31:15.211965 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:15.211777    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f249ed3-acb4-4e5d-b189-de21af9f1a25-openshift-service-ca\") pod \"maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w\" (UID: \"5f249ed3-acb4-4e5d-b189-de21af9f1a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w"
Sep 11 17:31:15.217261 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:15.217202    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w"]
Sep 11 17:31:15.300111 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:15.300084    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq"]
Sep 11 17:31:15.303303 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:31:15.303275    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd26d755_6e39_49b6_b956_98d5709b58b0.slice/crio-09e8d14b38a04b46590eaf8a48286681bde56ac37b67af7c5a799fabe075db78 WatchSource:0}: Error finding container 09e8d14b38a04b46590eaf8a48286681bde56ac37b67af7c5a799fabe075db78: Status 404 returned error can't find the container with id 09e8d14b38a04b46590eaf8a48286681bde56ac37b67af7c5a799fabe075db78
Sep 11 17:31:15.305277 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:15.305255    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 17:31:15.312293 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:15.312268    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkw4l\" (UniqueName: \"kubernetes.io/projected/5f249ed3-acb4-4e5d-b189-de21af9f1a25-kube-api-access-tkw4l\") pod \"maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w\" (UID: \"5f249ed3-acb4-4e5d-b189-de21af9f1a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w"
Sep 11 17:31:15.312391 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:15.312309    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f249ed3-acb4-4e5d-b189-de21af9f1a25-metrics-certs\") pod \"maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w\" (UID: \"5f249ed3-acb4-4e5d-b189-de21af9f1a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w"
Sep 11 17:31:15.312391 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:15.312362    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f249ed3-acb4-4e5d-b189-de21af9f1a25-maas-api-tls\") pod \"maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w\" (UID: \"5f249ed3-acb4-4e5d-b189-de21af9f1a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w"
Sep 11 17:31:15.312500 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:15.312398    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f249ed3-acb4-4e5d-b189-de21af9f1a25-openshift-service-ca\") pod \"maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w\" (UID: \"5f249ed3-acb4-4e5d-b189-de21af9f1a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w"
Sep 11 17:31:15.312500 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:31:15.312472    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-08c6b2-serving-cert: secret "maas-api-e2e-inf-a-08c6b2-serving-cert" not found
Sep 11 17:31:15.312564 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:31:15.312538    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f249ed3-acb4-4e5d-b189-de21af9f1a25-maas-api-tls podName:5f249ed3-acb4-4e5d-b189-de21af9f1a25 nodeName:}" failed. No retries permitted until 2026-09-11 17:31:15.812519213 +0000 UTC m=+756.206436653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5f249ed3-acb4-4e5d-b189-de21af9f1a25-maas-api-tls") pod "maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w" (UID: "5f249ed3-acb4-4e5d-b189-de21af9f1a25") : secret "maas-api-e2e-inf-a-08c6b2-serving-cert" not found
Sep 11 17:31:15.313157 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:15.313131    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f249ed3-acb4-4e5d-b189-de21af9f1a25-openshift-service-ca\") pod \"maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w\" (UID: \"5f249ed3-acb4-4e5d-b189-de21af9f1a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w"
Sep 11 17:31:15.314662 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:15.314644    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f249ed3-acb4-4e5d-b189-de21af9f1a25-metrics-certs\") pod \"maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w\" (UID: \"5f249ed3-acb4-4e5d-b189-de21af9f1a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w"
Sep 11 17:31:15.335570 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:15.335530    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkw4l\" (UniqueName: \"kubernetes.io/projected/5f249ed3-acb4-4e5d-b189-de21af9f1a25-kube-api-access-tkw4l\") pod \"maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w\" (UID: \"5f249ed3-acb4-4e5d-b189-de21af9f1a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w"
Sep 11 17:31:15.818527 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:15.818493    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f249ed3-acb4-4e5d-b189-de21af9f1a25-maas-api-tls\") pod \"maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w\" (UID: \"5f249ed3-acb4-4e5d-b189-de21af9f1a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w"
Sep 11 17:31:15.820856 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:15.820835    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f249ed3-acb4-4e5d-b189-de21af9f1a25-maas-api-tls\") pod \"maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w\" (UID: \"5f249ed3-acb4-4e5d-b189-de21af9f1a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w"
Sep 11 17:31:16.066833 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:16.066803    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-08c6b2-6df7cb9cd-q9mx5"]
Sep 11 17:31:16.071346 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:16.071292    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08c6b2-6df7cb9cd-q9mx5"
Sep 11 17:31:16.073407 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:16.073383    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-08c6b2-dockercfg-2dnf7\""
Sep 11 17:31:16.073514 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:16.073424    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-08c6b2\""
Sep 11 17:31:16.082910 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:16.082884    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-08c6b2-6df7cb9cd-q9mx5"]
Sep 11 17:31:16.119228 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:16.119189    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w"
Sep 11 17:31:16.127848 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:16.122510    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gqzg\" (UniqueName: \"kubernetes.io/projected/8d74b25a-7fd4-489c-943b-8fd444b32f38-kube-api-access-8gqzg\") pod \"payload-pre-processing-e2e-inf-a-08c6b2-6df7cb9cd-q9mx5\" (UID: \"8d74b25a-7fd4-489c-943b-8fd444b32f38\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08c6b2-6df7cb9cd-q9mx5"
Sep 11 17:31:16.127848 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:16.122583    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d74b25a-7fd4-489c-943b-8fd444b32f38-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-08c6b2-6df7cb9cd-q9mx5\" (UID: \"8d74b25a-7fd4-489c-943b-8fd444b32f38\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08c6b2-6df7cb9cd-q9mx5"
Sep 11 17:31:16.159580 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:16.159532    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq" event={"ID":"dd26d755-6e39-49b6-b956-98d5709b58b0","Type":"ContainerStarted","Data":"09e8d14b38a04b46590eaf8a48286681bde56ac37b67af7c5a799fabe075db78"}
Sep 11 17:31:16.224148 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:16.223915    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d74b25a-7fd4-489c-943b-8fd444b32f38-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-08c6b2-6df7cb9cd-q9mx5\" (UID: \"8d74b25a-7fd4-489c-943b-8fd444b32f38\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08c6b2-6df7cb9cd-q9mx5"
Sep 11 17:31:16.224148 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:16.224067    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gqzg\" (UniqueName: \"kubernetes.io/projected/8d74b25a-7fd4-489c-943b-8fd444b32f38-kube-api-access-8gqzg\") pod \"payload-pre-processing-e2e-inf-a-08c6b2-6df7cb9cd-q9mx5\" (UID: \"8d74b25a-7fd4-489c-943b-8fd444b32f38\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08c6b2-6df7cb9cd-q9mx5"
Sep 11 17:31:16.225085 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:16.225055    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d74b25a-7fd4-489c-943b-8fd444b32f38-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-08c6b2-6df7cb9cd-q9mx5\" (UID: \"8d74b25a-7fd4-489c-943b-8fd444b32f38\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08c6b2-6df7cb9cd-q9mx5"
Sep 11 17:31:16.236207 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:16.236172    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gqzg\" (UniqueName: \"kubernetes.io/projected/8d74b25a-7fd4-489c-943b-8fd444b32f38-kube-api-access-8gqzg\") pod \"payload-pre-processing-e2e-inf-a-08c6b2-6df7cb9cd-q9mx5\" (UID: \"8d74b25a-7fd4-489c-943b-8fd444b32f38\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08c6b2-6df7cb9cd-q9mx5"
Sep 11 17:31:16.382920 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:16.382881    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08c6b2-6df7cb9cd-q9mx5"
Sep 11 17:31:16.422466 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:16.422343    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w"]
Sep 11 17:31:16.529454 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:16.521780    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz"]
Sep 11 17:31:16.529454 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:16.527919    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz"
Sep 11 17:31:16.530444 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:16.530420    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-c7d192-serving-cert\""
Sep 11 17:31:16.538377 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:16.538346    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz"]
Sep 11 17:31:16.631026 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:16.630976    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/465d1839-c244-466a-9dca-c9ff8a9ab66a-maas-api-tls\") pod \"maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz\" (UID: \"465d1839-c244-466a-9dca-c9ff8a9ab66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz"
Sep 11 17:31:16.631280 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:16.631254    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkrmq\" (UniqueName: \"kubernetes.io/projected/465d1839-c244-466a-9dca-c9ff8a9ab66a-kube-api-access-hkrmq\") pod \"maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz\" (UID: \"465d1839-c244-466a-9dca-c9ff8a9ab66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz"
Sep 11 17:31:16.631427 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:16.631292    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/465d1839-c244-466a-9dca-c9ff8a9ab66a-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz\" (UID: \"465d1839-c244-466a-9dca-c9ff8a9ab66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz"
Sep 11 17:31:16.631427 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:16.631316    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/465d1839-c244-466a-9dca-c9ff8a9ab66a-metrics-certs\") pod \"maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz\" (UID: \"465d1839-c244-466a-9dca-c9ff8a9ab66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz"
Sep 11 17:31:16.690505 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:16.690415    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-08c6b2-6df7cb9cd-q9mx5"]
Sep 11 17:31:16.694867 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:31:16.694826    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d74b25a_7fd4_489c_943b_8fd444b32f38.slice/crio-9f11497f2f6b42202ce9ac5ea6714ccfa3be86022e20f6e5e0f9cfde35366555 WatchSource:0}: Error finding container 9f11497f2f6b42202ce9ac5ea6714ccfa3be86022e20f6e5e0f9cfde35366555: Status 404 returned error can't find the container with id 9f11497f2f6b42202ce9ac5ea6714ccfa3be86022e20f6e5e0f9cfde35366555
Sep 11 17:31:16.732185 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:16.732160    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/465d1839-c244-466a-9dca-c9ff8a9ab66a-maas-api-tls\") pod \"maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz\" (UID: \"465d1839-c244-466a-9dca-c9ff8a9ab66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz"
Sep 11 17:31:16.732322 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:16.732192    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkrmq\" (UniqueName: \"kubernetes.io/projected/465d1839-c244-466a-9dca-c9ff8a9ab66a-kube-api-access-hkrmq\") pod \"maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz\" (UID: \"465d1839-c244-466a-9dca-c9ff8a9ab66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz"
Sep 11 17:31:16.732322 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:16.732211    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/465d1839-c244-466a-9dca-c9ff8a9ab66a-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz\" (UID: \"465d1839-c244-466a-9dca-c9ff8a9ab66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz"
Sep 11 17:31:16.732322 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:16.732229    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/465d1839-c244-466a-9dca-c9ff8a9ab66a-metrics-certs\") pod \"maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz\" (UID: \"465d1839-c244-466a-9dca-c9ff8a9ab66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz"
Sep 11 17:31:16.732470 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:31:16.732325    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-c7d192-serving-cert: secret "maas-api-e2e-worker-w0-c7d192-serving-cert" not found
Sep 11 17:31:16.732470 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:31:16.732403    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/465d1839-c244-466a-9dca-c9ff8a9ab66a-maas-api-tls podName:465d1839-c244-466a-9dca-c9ff8a9ab66a nodeName:}" failed. No retries permitted until 2026-09-11 17:31:17.232381293 +0000 UTC m=+757.626298740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/465d1839-c244-466a-9dca-c9ff8a9ab66a-maas-api-tls") pod "maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz" (UID: "465d1839-c244-466a-9dca-c9ff8a9ab66a") : secret "maas-api-e2e-worker-w0-c7d192-serving-cert" not found
Sep 11 17:31:16.732892 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:16.732870    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/465d1839-c244-466a-9dca-c9ff8a9ab66a-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz\" (UID: \"465d1839-c244-466a-9dca-c9ff8a9ab66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz"
Sep 11 17:31:16.734966 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:16.734943    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/465d1839-c244-466a-9dca-c9ff8a9ab66a-metrics-certs\") pod \"maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz\" (UID: \"465d1839-c244-466a-9dca-c9ff8a9ab66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz"
Sep 11 17:31:16.740107 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:16.740073    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkrmq\" (UniqueName: \"kubernetes.io/projected/465d1839-c244-466a-9dca-c9ff8a9ab66a-kube-api-access-hkrmq\") pod \"maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz\" (UID: \"465d1839-c244-466a-9dca-c9ff8a9ab66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz"
Sep 11 17:31:17.171781 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:17.171101    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq" event={"ID":"dd26d755-6e39-49b6-b956-98d5709b58b0","Type":"ContainerStarted","Data":"ade2848bc4fb1509e4ba34f9270f68bafdaf92e451434a833ccb5987ee424d6e"}
Sep 11 17:31:17.172077 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:17.172028    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq"
Sep 11 17:31:17.174592 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:17.174457    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08c6b2-6df7cb9cd-q9mx5" event={"ID":"8d74b25a-7fd4-489c-943b-8fd444b32f38","Type":"ContainerStarted","Data":"a7c36504d070c824123abc943d8e516098e8cef56f089f8e2ebd22fb4eb58a21"}
Sep 11 17:31:17.174592 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:17.174513    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08c6b2-6df7cb9cd-q9mx5" event={"ID":"8d74b25a-7fd4-489c-943b-8fd444b32f38","Type":"ContainerStarted","Data":"9f11497f2f6b42202ce9ac5ea6714ccfa3be86022e20f6e5e0f9cfde35366555"}
Sep 11 17:31:17.174592 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:17.174559    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08c6b2-6df7cb9cd-q9mx5"
Sep 11 17:31:17.177091 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:17.177060    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w" event={"ID":"5f249ed3-acb4-4e5d-b189-de21af9f1a25","Type":"ContainerStarted","Data":"6988d1aeb216e89cc487af686998a28876712a9c1308f80e9b9e1d813bc3c5c0"}
Sep 11 17:31:17.177200 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:17.177171    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w" event={"ID":"5f249ed3-acb4-4e5d-b189-de21af9f1a25","Type":"ContainerStarted","Data":"bb230af9813967f766c16e8ef5f4e10edd7256c34c3083cbab9267d154011e1b"}
Sep 11 17:31:17.177711 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:17.177689    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w"
Sep 11 17:31:17.201276 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:17.201219    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq" podStartSLOduration=1.748870128 podStartE2EDuration="3.201198189s" totalImagesPullingTime="1.452328061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:31:17.201028924 +0000 UTC m=+757.594946393" watchObservedRunningTime="2026-09-11 17:31:17.201198189 +0000 UTC m=+757.595115645"
Sep 11 17:31:17.227679 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:17.227624    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w" podStartSLOduration=1.8666583810000001 podStartE2EDuration="2.227605824s" totalImagesPullingTime="360.947443ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:31:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:31:17.225217816 +0000 UTC m=+757.619135272" watchObservedRunningTime="2026-09-11 17:31:17.227605824 +0000 UTC m=+757.621523279"
Sep 11 17:31:17.237432 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:17.237141    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/465d1839-c244-466a-9dca-c9ff8a9ab66a-maas-api-tls\") pod \"maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz\" (UID: \"465d1839-c244-466a-9dca-c9ff8a9ab66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz"
Sep 11 17:31:17.237432 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:31:17.237329    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-c7d192-serving-cert: secret "maas-api-e2e-worker-w0-c7d192-serving-cert" not found
Sep 11 17:31:17.237432 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:31:17.237398    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/465d1839-c244-466a-9dca-c9ff8a9ab66a-maas-api-tls podName:465d1839-c244-466a-9dca-c9ff8a9ab66a nodeName:}" failed. No retries permitted until 2026-09-11 17:31:18.237379031 +0000 UTC m=+758.631296477 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/465d1839-c244-466a-9dca-c9ff8a9ab66a-maas-api-tls") pod "maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz" (UID: "465d1839-c244-466a-9dca-c9ff8a9ab66a") : secret "maas-api-e2e-worker-w0-c7d192-serving-cert" not found
Sep 11 17:31:17.250603 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:17.250555    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08c6b2-6df7cb9cd-q9mx5" podStartSLOduration=1.250537478 podStartE2EDuration="1.250537478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:31:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:31:17.249813886 +0000 UTC m=+757.643731344" watchObservedRunningTime="2026-09-11 17:31:17.250537478 +0000 UTC m=+757.644454935"
Sep 11 17:31:18.246689 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:18.246652    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/465d1839-c244-466a-9dca-c9ff8a9ab66a-maas-api-tls\") pod \"maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz\" (UID: \"465d1839-c244-466a-9dca-c9ff8a9ab66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz"
Sep 11 17:31:18.249370 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:18.249345    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/465d1839-c244-466a-9dca-c9ff8a9ab66a-maas-api-tls\") pod \"maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz\" (UID: \"465d1839-c244-466a-9dca-c9ff8a9ab66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz"
Sep 11 17:31:18.353628 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:18.353587    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz"
Sep 11 17:31:18.627425 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:18.627347    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz"]
Sep 11 17:31:18.630239 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:31:18.630214    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod465d1839_c244_466a_9dca_c9ff8a9ab66a.slice/crio-18fe0ef77efc9142363adabf89441771caa8d0157334fc5ebbeda4de2dc6a544 WatchSource:0}: Error finding container 18fe0ef77efc9142363adabf89441771caa8d0157334fc5ebbeda4de2dc6a544: Status 404 returned error can't find the container with id 18fe0ef77efc9142363adabf89441771caa8d0157334fc5ebbeda4de2dc6a544
Sep 11 17:31:19.190268 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:19.190235    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz" event={"ID":"465d1839-c244-466a-9dca-c9ff8a9ab66a","Type":"ContainerStarted","Data":"de7711a520970670a6a1bd1d54b15373ad9a8ddeb0fd68e8513e75a08651bd4f"}
Sep 11 17:31:19.190459 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:19.190286    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz" event={"ID":"465d1839-c244-466a-9dca-c9ff8a9ab66a","Type":"ContainerStarted","Data":"18fe0ef77efc9142363adabf89441771caa8d0157334fc5ebbeda4de2dc6a544"}
Sep 11 17:31:19.209950 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:19.209903    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz" podStartSLOduration=2.834874615 podStartE2EDuration="3.209885606s" totalImagesPullingTime="375.010991ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:31:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:31:19.20846851 +0000 UTC m=+759.602385977" watchObservedRunningTime="2026-09-11 17:31:19.209885606 +0000 UTC m=+759.603803086"
Sep 11 17:31:20.196244 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:20.196212    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz"
Sep 11 17:31:24.034835 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:24.034795    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh"]
Sep 11 17:31:24.038927 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:24.038903    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh"
Sep 11 17:31:24.041193 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:24.041172    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-23a9cc4a-serving-cert\""
Sep 11 17:31:24.057344 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:24.057311    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh"]
Sep 11 17:31:24.104136 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:24.104100    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9-maas-api-tls\") pod \"maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh\" (UID: \"510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh"
Sep 11 17:31:24.104315 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:24.104157    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9-openshift-service-ca\") pod \"maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh\" (UID: \"510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh"
Sep 11 17:31:24.104315 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:24.104188    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9-metrics-certs\") pod \"maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh\" (UID: \"510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh"
Sep 11 17:31:24.104315 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:24.104268    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt72b\" (UniqueName: \"kubernetes.io/projected/510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9-kube-api-access-wt72b\") pod \"maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh\" (UID: \"510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh"
Sep 11 17:31:24.195778 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:24.195744    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq"
Sep 11 17:31:24.196228 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:24.196207    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w"
Sep 11 17:31:24.205063 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:24.205043    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt72b\" (UniqueName: \"kubernetes.io/projected/510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9-kube-api-access-wt72b\") pod \"maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh\" (UID: \"510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh"
Sep 11 17:31:24.205136 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:24.205092    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9-maas-api-tls\") pod \"maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh\" (UID: \"510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh"
Sep 11 17:31:24.205211 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:24.205185    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9-openshift-service-ca\") pod \"maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh\" (UID: \"510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh"
Sep 11 17:31:24.205211 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:24.205212    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9-metrics-certs\") pod \"maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh\" (UID: \"510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh"
Sep 11 17:31:24.205413 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:31:24.205269    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-23a9cc4a-serving-cert: secret "maas-api-e2e-ait-23a9cc4a-serving-cert" not found
Sep 11 17:31:24.205413 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:31:24.205331    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9-maas-api-tls podName:510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9 nodeName:}" failed. No retries permitted until 2026-09-11 17:31:24.705312778 +0000 UTC m=+765.099230210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9-maas-api-tls") pod "maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh" (UID: "510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9") : secret "maas-api-e2e-ait-23a9cc4a-serving-cert" not found
Sep 11 17:31:24.205878 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:24.205851    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9-openshift-service-ca\") pod \"maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh\" (UID: \"510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh"
Sep 11 17:31:24.207617 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:24.207599    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9-metrics-certs\") pod \"maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh\" (UID: \"510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh"
Sep 11 17:31:24.222600 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:24.222581    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt72b\" (UniqueName: \"kubernetes.io/projected/510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9-kube-api-access-wt72b\") pod \"maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh\" (UID: \"510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh"
Sep 11 17:31:24.709573 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:24.709535    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9-maas-api-tls\") pod \"maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh\" (UID: \"510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh"
Sep 11 17:31:24.712053 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:24.712023    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9-maas-api-tls\") pod \"maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh\" (UID: \"510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh"
Sep 11 17:31:24.882223 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:24.882187    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-23a9cc4a-657f8fcd78-gppbh"]
Sep 11 17:31:24.885773 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:24.885755    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-23a9cc4a-657f8fcd78-gppbh"
Sep 11 17:31:24.889168 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:24.889142    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-23a9cc4a-dockercfg-4gsxn\""
Sep 11 17:31:24.894168 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:24.894147    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-23a9cc4a\""
Sep 11 17:31:24.961471 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:24.961383    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-23a9cc4a-657f8fcd78-gppbh"]
Sep 11 17:31:24.975084 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:24.974948    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh"
Sep 11 17:31:25.011949 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:25.011910    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872be239-eb78-45dd-a06d-06fa3f288989-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-23a9cc4a-657f8fcd78-gppbh\" (UID: \"872be239-eb78-45dd-a06d-06fa3f288989\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-23a9cc4a-657f8fcd78-gppbh"
Sep 11 17:31:25.012118 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:25.012040    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhcv5\" (UniqueName: \"kubernetes.io/projected/872be239-eb78-45dd-a06d-06fa3f288989-kube-api-access-rhcv5\") pod \"payload-pre-processing-e2e-ait-23a9cc4a-657f8fcd78-gppbh\" (UID: \"872be239-eb78-45dd-a06d-06fa3f288989\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-23a9cc4a-657f8fcd78-gppbh"
Sep 11 17:31:25.112760 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:25.112718    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872be239-eb78-45dd-a06d-06fa3f288989-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-23a9cc4a-657f8fcd78-gppbh\" (UID: \"872be239-eb78-45dd-a06d-06fa3f288989\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-23a9cc4a-657f8fcd78-gppbh"
Sep 11 17:31:25.113241 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:25.112814    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhcv5\" (UniqueName: \"kubernetes.io/projected/872be239-eb78-45dd-a06d-06fa3f288989-kube-api-access-rhcv5\") pod \"payload-pre-processing-e2e-ait-23a9cc4a-657f8fcd78-gppbh\" (UID: \"872be239-eb78-45dd-a06d-06fa3f288989\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-23a9cc4a-657f8fcd78-gppbh"
Sep 11 17:31:25.113533 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:25.113508    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872be239-eb78-45dd-a06d-06fa3f288989-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-23a9cc4a-657f8fcd78-gppbh\" (UID: \"872be239-eb78-45dd-a06d-06fa3f288989\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-23a9cc4a-657f8fcd78-gppbh"
Sep 11 17:31:25.141580 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:25.141548    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhcv5\" (UniqueName: \"kubernetes.io/projected/872be239-eb78-45dd-a06d-06fa3f288989-kube-api-access-rhcv5\") pod \"payload-pre-processing-e2e-ait-23a9cc4a-657f8fcd78-gppbh\" (UID: \"872be239-eb78-45dd-a06d-06fa3f288989\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-23a9cc4a-657f8fcd78-gppbh"
Sep 11 17:31:25.195804 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:25.195768    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-23a9cc4a-657f8fcd78-gppbh"
Sep 11 17:31:25.254437 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:25.254358    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh"]
Sep 11 17:31:25.504874 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:25.504850    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-23a9cc4a-657f8fcd78-gppbh"]
Sep 11 17:31:25.506989 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:31:25.506962    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod872be239_eb78_45dd_a06d_06fa3f288989.slice/crio-865484217ee64cadcf265d8012e8fb44226efe214bcd3d8c10e2c1b68cd764e6 WatchSource:0}: Error finding container 865484217ee64cadcf265d8012e8fb44226efe214bcd3d8c10e2c1b68cd764e6: Status 404 returned error can't find the container with id 865484217ee64cadcf265d8012e8fb44226efe214bcd3d8c10e2c1b68cd764e6
Sep 11 17:31:26.205947 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:26.205921    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz"
Sep 11 17:31:26.225596 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:26.225478    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-23a9cc4a-657f8fcd78-gppbh" event={"ID":"872be239-eb78-45dd-a06d-06fa3f288989","Type":"ContainerStarted","Data":"8d010cd8d3721677966e346fdf1f11109acfc3b1d61b5a8046f237522ccdcf31"}
Sep 11 17:31:26.225596 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:26.225550    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-23a9cc4a-657f8fcd78-gppbh" event={"ID":"872be239-eb78-45dd-a06d-06fa3f288989","Type":"ContainerStarted","Data":"865484217ee64cadcf265d8012e8fb44226efe214bcd3d8c10e2c1b68cd764e6"}
Sep 11 17:31:26.225596 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:26.225592    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-23a9cc4a-657f8fcd78-gppbh"
Sep 11 17:31:26.227557 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:26.227529    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh" event={"ID":"510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9","Type":"ContainerStarted","Data":"fbcf24d926ff855701676a872c101861173770fb6ef84c98070d81a1c3abf6bd"}
Sep 11 17:31:26.227658 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:26.227578    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh" event={"ID":"510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9","Type":"ContainerStarted","Data":"c777acbb3de2bed10b1fa90a227aae35907bf882f23af39d647b1f59b93a56e5"}
Sep 11 17:31:26.227658 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:26.227627    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh"
Sep 11 17:31:26.247027 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:26.243296    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-23a9cc4a-657f8fcd78-gppbh" podStartSLOduration=2.243274481 podStartE2EDuration="2.243274481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:31:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:31:26.241221507 +0000 UTC m=+766.635138991" watchObservedRunningTime="2026-09-11 17:31:26.243274481 +0000 UTC m=+766.637191937"
Sep 11 17:31:26.284836 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:26.284701    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh" podStartSLOduration=1.940993088 podStartE2EDuration="2.284680642s" totalImagesPullingTime="343.687554ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:31:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:31:26.284319015 +0000 UTC m=+766.678236471" watchObservedRunningTime="2026-09-11 17:31:26.284680642 +0000 UTC m=+766.678598097"
Sep 11 17:31:27.264028 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:27.261420    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-49e398-567f8655bf-qf2pj"]
Sep 11 17:31:27.287433 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:27.286708    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-49e398-567f8655bf-qf2pj"]
Sep 11 17:31:27.287433 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:27.286839    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-49e398-567f8655bf-qf2pj"
Sep 11 17:31:27.291228 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:27.291019    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-49e398-dockercfg-8rmm7\""
Sep 11 17:31:27.291228 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:27.291021    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-49e398\""
Sep 11 17:31:27.331585 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:27.331555    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1c624d-7469-4934-9214-fc400af5de05-plugins-config-volume\") pod \"payload-processing-e2e-autores-49e398-567f8655bf-qf2pj\" (UID: \"6c1c624d-7469-4934-9214-fc400af5de05\") " pod="openshift-ingress/payload-processing-e2e-autores-49e398-567f8655bf-qf2pj"
Sep 11 17:31:27.331711 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:27.331633    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4plxs\" (UniqueName: \"kubernetes.io/projected/6c1c624d-7469-4934-9214-fc400af5de05-kube-api-access-4plxs\") pod \"payload-processing-e2e-autores-49e398-567f8655bf-qf2pj\" (UID: \"6c1c624d-7469-4934-9214-fc400af5de05\") " pod="openshift-ingress/payload-processing-e2e-autores-49e398-567f8655bf-qf2pj"
Sep 11 17:31:27.432542 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:27.432507    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1c624d-7469-4934-9214-fc400af5de05-plugins-config-volume\") pod \"payload-processing-e2e-autores-49e398-567f8655bf-qf2pj\" (UID: \"6c1c624d-7469-4934-9214-fc400af5de05\") " pod="openshift-ingress/payload-processing-e2e-autores-49e398-567f8655bf-qf2pj"
Sep 11 17:31:27.432745 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:27.432566    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4plxs\" (UniqueName: \"kubernetes.io/projected/6c1c624d-7469-4934-9214-fc400af5de05-kube-api-access-4plxs\") pod \"payload-processing-e2e-autores-49e398-567f8655bf-qf2pj\" (UID: \"6c1c624d-7469-4934-9214-fc400af5de05\") " pod="openshift-ingress/payload-processing-e2e-autores-49e398-567f8655bf-qf2pj"
Sep 11 17:31:27.433461 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:27.433434    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1c624d-7469-4934-9214-fc400af5de05-plugins-config-volume\") pod \"payload-processing-e2e-autores-49e398-567f8655bf-qf2pj\" (UID: \"6c1c624d-7469-4934-9214-fc400af5de05\") " pod="openshift-ingress/payload-processing-e2e-autores-49e398-567f8655bf-qf2pj"
Sep 11 17:31:27.441151 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:27.441117    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4plxs\" (UniqueName: \"kubernetes.io/projected/6c1c624d-7469-4934-9214-fc400af5de05-kube-api-access-4plxs\") pod \"payload-processing-e2e-autores-49e398-567f8655bf-qf2pj\" (UID: \"6c1c624d-7469-4934-9214-fc400af5de05\") " pod="openshift-ingress/payload-processing-e2e-autores-49e398-567f8655bf-qf2pj"
Sep 11 17:31:27.601123 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:27.601075    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-49e398-567f8655bf-qf2pj"
Sep 11 17:31:27.905047 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:27.904869    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-49e398-567f8655bf-qf2pj"]
Sep 11 17:31:27.907840 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:31:27.907810    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c1c624d_7469_4934_9214_fc400af5de05.slice/crio-02aa6436c329e944b9c05c9ecc868e061feafdd9d79ae98df23747cbd622bf3c WatchSource:0}: Error finding container 02aa6436c329e944b9c05c9ecc868e061feafdd9d79ae98df23747cbd622bf3c: Status 404 returned error can't find the container with id 02aa6436c329e944b9c05c9ecc868e061feafdd9d79ae98df23747cbd622bf3c
Sep 11 17:31:28.185192 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:28.185100    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08c6b2-6df7cb9cd-q9mx5"
Sep 11 17:31:28.240707 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:28.240674    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-49e398-567f8655bf-qf2pj" event={"ID":"6c1c624d-7469-4934-9214-fc400af5de05","Type":"ContainerStarted","Data":"8a7aa1820b48d5dda5e03f9aeaf3e37246313e7bb1348a2456fd185fa683af45"}
Sep 11 17:31:28.240903 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:28.240733    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-49e398-567f8655bf-qf2pj" event={"ID":"6c1c624d-7469-4934-9214-fc400af5de05","Type":"ContainerStarted","Data":"02aa6436c329e944b9c05c9ecc868e061feafdd9d79ae98df23747cbd622bf3c"}
Sep 11 17:31:28.240903 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:28.240816    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-49e398-567f8655bf-qf2pj"
Sep 11 17:31:28.259727 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:28.259666    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-49e398-567f8655bf-qf2pj" podStartSLOduration=1.2596450240000001 podStartE2EDuration="1.259645024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:31:28.256747559 +0000 UTC m=+768.650665011" watchObservedRunningTime="2026-09-11 17:31:28.259645024 +0000 UTC m=+768.653562485"
Sep 11 17:31:31.259598 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:31.259558    2562 generic.go:356] "Generic (PLEG): container finished" podID="00b26df4-2ff6-45f3-b020-e1854b2060d6" containerID="22f018c6731d122966eae74773279a0812791889fc4c6b848a8f54a8a32d1332" exitCode=0
Sep 11 17:31:31.259979 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:31.259634    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4" event={"ID":"00b26df4-2ff6-45f3-b020-e1854b2060d6","Type":"ContainerDied","Data":"22f018c6731d122966eae74773279a0812791889fc4c6b848a8f54a8a32d1332"}
Sep 11 17:31:31.260077 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:31.260062    2562 scope.go:117] "RemoveContainer" containerID="22f018c6731d122966eae74773279a0812791889fc4c6b848a8f54a8a32d1332"
Sep 11 17:31:31.261237 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:31.261212    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:31:31.261317 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:31.261276    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:31:31.261317 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:31.261307    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:31:32.237964 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:32.237933    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh"
Sep 11 17:31:32.270704 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:32.269917    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4" event={"ID":"00b26df4-2ff6-45f3-b020-e1854b2060d6","Type":"ContainerStarted","Data":"6476d3b8dc311012ba5147baaa84da12977b4c6f9e384d3472b6e970611bfbc8"}
Sep 11 17:31:32.882533 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:32.882490    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"
Sep 11 17:31:32.882818 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:32.882792    2562 patch_prober.go:28] interesting pod/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 11 17:31:32.882952 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:32.882852    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4" podUID="00b26df4-2ff6-45f3-b020-e1854b2060d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 11 17:31:33.882497 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:33.882455    2562 patch_prober.go:28] interesting pod/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 11 17:31:33.882982 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:33.882523    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4" podUID="00b26df4-2ff6-45f3-b020-e1854b2060d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 11 17:31:34.882981 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:34.882944    2562 patch_prober.go:28] interesting pod/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 11 17:31:34.883403 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:34.883033    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4" podUID="00b26df4-2ff6-45f3-b020-e1854b2060d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 11 17:31:35.882859 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:35.882817    2562 patch_prober.go:28] interesting pod/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 11 17:31:35.883078 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:35.882884    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4" podUID="00b26df4-2ff6-45f3-b020-e1854b2060d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 11 17:31:36.883203 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:36.883163    2562 patch_prober.go:28] interesting pod/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 11 17:31:36.883626 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:36.883232    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4" podUID="00b26df4-2ff6-45f3-b020-e1854b2060d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 11 17:31:37.235057 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:37.234951    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-23a9cc4a-657f8fcd78-gppbh"
Sep 11 17:31:37.883198 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:37.883153    2562 patch_prober.go:28] interesting pod/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 11 17:31:37.883411 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:37.883220    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4" podUID="00b26df4-2ff6-45f3-b020-e1854b2060d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 11 17:31:38.886120 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:38.886088    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"
Sep 11 17:31:38.886574 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:38.886365    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"
Sep 11 17:31:38.887179 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:38.887162    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"
Sep 11 17:31:39.249824 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:39.249733    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-49e398-567f8655bf-qf2pj"
Sep 11 17:31:45.142937 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:45.142900    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf"]
Sep 11 17:31:45.151518 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:45.150934    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf"
Sep 11 17:31:45.153302 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:45.153275    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-cbbb09-serving-cert\""
Sep 11 17:31:45.156978 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:45.156949    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf"]
Sep 11 17:31:45.301065 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:45.301034    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a57696c0-abc7-4258-896c-476008734dfe-metrics-certs\") pod \"maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf\" (UID: \"a57696c0-abc7-4258-896c-476008734dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf"
Sep 11 17:31:45.301248 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:45.301147    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a57696c0-abc7-4258-896c-476008734dfe-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf\" (UID: \"a57696c0-abc7-4258-896c-476008734dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf"
Sep 11 17:31:45.301248 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:45.301174    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsx89\" (UniqueName: \"kubernetes.io/projected/a57696c0-abc7-4258-896c-476008734dfe-kube-api-access-hsx89\") pod \"maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf\" (UID: \"a57696c0-abc7-4258-896c-476008734dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf"
Sep 11 17:31:45.301248 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:45.301200    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a57696c0-abc7-4258-896c-476008734dfe-maas-api-tls\") pod \"maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf\" (UID: \"a57696c0-abc7-4258-896c-476008734dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf"
Sep 11 17:31:45.402327 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:45.402229    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a57696c0-abc7-4258-896c-476008734dfe-metrics-certs\") pod \"maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf\" (UID: \"a57696c0-abc7-4258-896c-476008734dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf"
Sep 11 17:31:45.402327 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:45.402314    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a57696c0-abc7-4258-896c-476008734dfe-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf\" (UID: \"a57696c0-abc7-4258-896c-476008734dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf"
Sep 11 17:31:45.402536 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:45.402334    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsx89\" (UniqueName: \"kubernetes.io/projected/a57696c0-abc7-4258-896c-476008734dfe-kube-api-access-hsx89\") pod \"maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf\" (UID: \"a57696c0-abc7-4258-896c-476008734dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf"
Sep 11 17:31:45.402536 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:45.402359    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a57696c0-abc7-4258-896c-476008734dfe-maas-api-tls\") pod \"maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf\" (UID: \"a57696c0-abc7-4258-896c-476008734dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf"
Sep 11 17:31:45.403145 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:45.403119    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a57696c0-abc7-4258-896c-476008734dfe-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf\" (UID: \"a57696c0-abc7-4258-896c-476008734dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf"
Sep 11 17:31:45.404812 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:45.404790    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a57696c0-abc7-4258-896c-476008734dfe-metrics-certs\") pod \"maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf\" (UID: \"a57696c0-abc7-4258-896c-476008734dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf"
Sep 11 17:31:45.404902 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:45.404861    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a57696c0-abc7-4258-896c-476008734dfe-maas-api-tls\") pod \"maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf\" (UID: \"a57696c0-abc7-4258-896c-476008734dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf"
Sep 11 17:31:45.409739 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:45.409718    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsx89\" (UniqueName: \"kubernetes.io/projected/a57696c0-abc7-4258-896c-476008734dfe-kube-api-access-hsx89\") pod \"maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf\" (UID: \"a57696c0-abc7-4258-896c-476008734dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf"
Sep 11 17:31:45.464132 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:45.464097    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf"
Sep 11 17:31:45.739794 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:45.739767    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf"]
Sep 11 17:31:45.742083 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:31:45.742050    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda57696c0_abc7_4258_896c_476008734dfe.slice/crio-dd5d55eb8fbba23c103c1abdf073eb9450fb63e7a512342d300c83d4eaeca397 WatchSource:0}: Error finding container dd5d55eb8fbba23c103c1abdf073eb9450fb63e7a512342d300c83d4eaeca397: Status 404 returned error can't find the container with id dd5d55eb8fbba23c103c1abdf073eb9450fb63e7a512342d300c83d4eaeca397
Sep 11 17:31:46.342801 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:46.342761    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf" event={"ID":"a57696c0-abc7-4258-896c-476008734dfe","Type":"ContainerStarted","Data":"b50f654e0f8f614a73cecfa57777e75be4dbe8f2451b7feb11f518846a2adcc5"}
Sep 11 17:31:46.343291 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:46.342820    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf" event={"ID":"a57696c0-abc7-4258-896c-476008734dfe","Type":"ContainerStarted","Data":"dd5d55eb8fbba23c103c1abdf073eb9450fb63e7a512342d300c83d4eaeca397"}
Sep 11 17:31:46.343291 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:46.342841    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf"
Sep 11 17:31:46.360460 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:46.360405    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf" podStartSLOduration=0.938179248 podStartE2EDuration="1.360385249s" totalImagesPullingTime="422.206001ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:31:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:31:46.359753565 +0000 UTC m=+786.753671019" watchObservedRunningTime="2026-09-11 17:31:46.360385249 +0000 UTC m=+786.754302707"
Sep 11 17:31:48.249175 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:48.249143    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh"]
Sep 11 17:31:48.249598 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:48.249379    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh" podUID="510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9" containerName="maas-api" containerID="cri-o://fbcf24d926ff855701676a872c101861173770fb6ef84c98070d81a1c3abf6bd" gracePeriod=30
Sep 11 17:31:48.312938 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:48.312905    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-23a9cc4a-657f8fcd78-gppbh"]
Sep 11 17:31:48.313155 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:48.313130    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-23a9cc4a-657f8fcd78-gppbh" podUID="872be239-eb78-45dd-a06d-06fa3f288989" containerName="payload-pre-processing" containerID="cri-o://8d010cd8d3721677966e346fdf1f11109acfc3b1d61b5a8046f237522ccdcf31" gracePeriod=30
Sep 11 17:31:48.582610 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:48.582578    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh"
Sep 11 17:31:48.605193 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:48.605118    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-23a9cc4a-657f8fcd78-gppbh"
Sep 11 17:31:48.636752 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:48.636721    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhcv5\" (UniqueName: \"kubernetes.io/projected/872be239-eb78-45dd-a06d-06fa3f288989-kube-api-access-rhcv5\") pod \"872be239-eb78-45dd-a06d-06fa3f288989\" (UID: \"872be239-eb78-45dd-a06d-06fa3f288989\") "
Sep 11 17:31:48.636905 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:48.636770    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9-metrics-certs\") pod \"510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9\" (UID: \"510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9\") "
Sep 11 17:31:48.636905 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:48.636826    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wt72b\" (UniqueName: \"kubernetes.io/projected/510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9-kube-api-access-wt72b\") pod \"510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9\" (UID: \"510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9\") "
Sep 11 17:31:48.636905 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:48.636861    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9-maas-api-tls\") pod \"510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9\" (UID: \"510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9\") "
Sep 11 17:31:48.636905 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:48.636892    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872be239-eb78-45dd-a06d-06fa3f288989-plugins-config-volume\") pod \"872be239-eb78-45dd-a06d-06fa3f288989\" (UID: \"872be239-eb78-45dd-a06d-06fa3f288989\") "
Sep 11 17:31:48.637158 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:48.636979    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9-openshift-service-ca\") pod \"510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9\" (UID: \"510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9\") "
Sep 11 17:31:48.637392 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:48.637361    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/872be239-eb78-45dd-a06d-06fa3f288989-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "872be239-eb78-45dd-a06d-06fa3f288989" (UID: "872be239-eb78-45dd-a06d-06fa3f288989"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:31:48.637484 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:48.637389    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9" (UID: "510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:31:48.639151 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:48.639126    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9" (UID: "510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:31:48.639267 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:48.639226    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9-kube-api-access-wt72b" (OuterVolumeSpecName: "kube-api-access-wt72b") pod "510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9" (UID: "510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9"). InnerVolumeSpecName "kube-api-access-wt72b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:31:48.639267 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:48.639243    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9" (UID: "510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:31:48.639345 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:48.639293    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/872be239-eb78-45dd-a06d-06fa3f288989-kube-api-access-rhcv5" (OuterVolumeSpecName: "kube-api-access-rhcv5") pod "872be239-eb78-45dd-a06d-06fa3f288989" (UID: "872be239-eb78-45dd-a06d-06fa3f288989"). InnerVolumeSpecName "kube-api-access-rhcv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:31:48.738078 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:48.738043    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9-openshift-service-ca\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:31:48.738078 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:48.738074    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhcv5\" (UniqueName: \"kubernetes.io/projected/872be239-eb78-45dd-a06d-06fa3f288989-kube-api-access-rhcv5\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:31:48.738078 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:48.738085    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9-metrics-certs\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:31:48.738313 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:48.738094    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt72b\" (UniqueName: \"kubernetes.io/projected/510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9-kube-api-access-wt72b\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:31:48.738313 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:48.738103    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9-maas-api-tls\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:31:48.738313 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:48.738111    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872be239-eb78-45dd-a06d-06fa3f288989-plugins-config-volume\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:31:49.359339 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:49.359303    2562 generic.go:356] "Generic (PLEG): container finished" podID="510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9" containerID="fbcf24d926ff855701676a872c101861173770fb6ef84c98070d81a1c3abf6bd" exitCode=0
Sep 11 17:31:49.359781 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:49.359396    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh" event={"ID":"510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9","Type":"ContainerDied","Data":"fbcf24d926ff855701676a872c101861173770fb6ef84c98070d81a1c3abf6bd"}
Sep 11 17:31:49.359781 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:49.359406    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh"
Sep 11 17:31:49.359781 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:49.359436    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh" event={"ID":"510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9","Type":"ContainerDied","Data":"c777acbb3de2bed10b1fa90a227aae35907bf882f23af39d647b1f59b93a56e5"}
Sep 11 17:31:49.359781 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:49.359458    2562 scope.go:117] "RemoveContainer" containerID="fbcf24d926ff855701676a872c101861173770fb6ef84c98070d81a1c3abf6bd"
Sep 11 17:31:49.361077 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:49.361049    2562 generic.go:356] "Generic (PLEG): container finished" podID="872be239-eb78-45dd-a06d-06fa3f288989" containerID="8d010cd8d3721677966e346fdf1f11109acfc3b1d61b5a8046f237522ccdcf31" exitCode=0
Sep 11 17:31:49.361163 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:49.361113    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-23a9cc4a-657f8fcd78-gppbh"
Sep 11 17:31:49.361163 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:49.361138    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-23a9cc4a-657f8fcd78-gppbh" event={"ID":"872be239-eb78-45dd-a06d-06fa3f288989","Type":"ContainerDied","Data":"8d010cd8d3721677966e346fdf1f11109acfc3b1d61b5a8046f237522ccdcf31"}
Sep 11 17:31:49.361234 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:49.361186    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-23a9cc4a-657f8fcd78-gppbh" event={"ID":"872be239-eb78-45dd-a06d-06fa3f288989","Type":"ContainerDied","Data":"865484217ee64cadcf265d8012e8fb44226efe214bcd3d8c10e2c1b68cd764e6"}
Sep 11 17:31:49.374039 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:49.368810    2562 scope.go:117] "RemoveContainer" containerID="fbcf24d926ff855701676a872c101861173770fb6ef84c98070d81a1c3abf6bd"
Sep 11 17:31:49.374039 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:31:49.369201    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbcf24d926ff855701676a872c101861173770fb6ef84c98070d81a1c3abf6bd\": container with ID starting with fbcf24d926ff855701676a872c101861173770fb6ef84c98070d81a1c3abf6bd not found: ID does not exist" containerID="fbcf24d926ff855701676a872c101861173770fb6ef84c98070d81a1c3abf6bd"
Sep 11 17:31:49.374039 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:49.369241    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbcf24d926ff855701676a872c101861173770fb6ef84c98070d81a1c3abf6bd"} err="failed to get container status \"fbcf24d926ff855701676a872c101861173770fb6ef84c98070d81a1c3abf6bd\": rpc error: code = NotFound desc = could not find container \"fbcf24d926ff855701676a872c101861173770fb6ef84c98070d81a1c3abf6bd\": container with ID starting with fbcf24d926ff855701676a872c101861173770fb6ef84c98070d81a1c3abf6bd not found: ID does not exist"
Sep 11 17:31:49.374039 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:49.369267    2562 scope.go:117] "RemoveContainer" containerID="8d010cd8d3721677966e346fdf1f11109acfc3b1d61b5a8046f237522ccdcf31"
Sep 11 17:31:49.381874 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:49.381858    2562 scope.go:117] "RemoveContainer" containerID="8d010cd8d3721677966e346fdf1f11109acfc3b1d61b5a8046f237522ccdcf31"
Sep 11 17:31:49.382144 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:31:49.382125    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d010cd8d3721677966e346fdf1f11109acfc3b1d61b5a8046f237522ccdcf31\": container with ID starting with 8d010cd8d3721677966e346fdf1f11109acfc3b1d61b5a8046f237522ccdcf31 not found: ID does not exist" containerID="8d010cd8d3721677966e346fdf1f11109acfc3b1d61b5a8046f237522ccdcf31"
Sep 11 17:31:49.382193 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:49.382152    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d010cd8d3721677966e346fdf1f11109acfc3b1d61b5a8046f237522ccdcf31"} err="failed to get container status \"8d010cd8d3721677966e346fdf1f11109acfc3b1d61b5a8046f237522ccdcf31\": rpc error: code = NotFound desc = could not find container \"8d010cd8d3721677966e346fdf1f11109acfc3b1d61b5a8046f237522ccdcf31\": container with ID starting with 8d010cd8d3721677966e346fdf1f11109acfc3b1d61b5a8046f237522ccdcf31 not found: ID does not exist"
Sep 11 17:31:49.392320 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:49.392299    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-23a9cc4a-657f8fcd78-gppbh"]
Sep 11 17:31:49.400026 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:49.399987    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-23a9cc4a-657f8fcd78-gppbh"]
Sep 11 17:31:49.413263 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:49.413242    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh"]
Sep 11 17:31:49.416800 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:49.416778    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-23a9cc4a-57bcb5f4d4-nsqjh"]
Sep 11 17:31:50.128108 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:50.128067    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9" path="/var/lib/kubelet/pods/510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9/volumes"
Sep 11 17:31:50.128631 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:50.128613    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="872be239-eb78-45dd-a06d-06fa3f288989" path="/var/lib/kubelet/pods/872be239-eb78-45dd-a06d-06fa3f288989/volumes"
Sep 11 17:31:52.354159 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:52.354132    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf"
Sep 11 17:31:54.805568 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:54.805529    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc"]
Sep 11 17:31:54.806299 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:54.806276    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9" containerName="maas-api"
Sep 11 17:31:54.806473 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:54.806450    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9" containerName="maas-api"
Sep 11 17:31:54.806593 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:54.806478    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="872be239-eb78-45dd-a06d-06fa3f288989" containerName="payload-pre-processing"
Sep 11 17:31:54.806593 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:54.806488    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="872be239-eb78-45dd-a06d-06fa3f288989" containerName="payload-pre-processing"
Sep 11 17:31:54.806695 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:54.806619    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="872be239-eb78-45dd-a06d-06fa3f288989" containerName="payload-pre-processing"
Sep 11 17:31:54.806695 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:54.806631    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="510d73d2-9d38-47d2-8b93-bc2a2a5b3aa9" containerName="maas-api"
Sep 11 17:31:54.809253 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:54.809231    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc"
Sep 11 17:31:54.812810 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:54.812787    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-d19de295-serving-cert\""
Sep 11 17:31:54.824028 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:54.824003    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc"]
Sep 11 17:31:54.894970 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:54.894935    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6203004-160a-49e6-bea5-6fe9e8dcdfea-maas-api-tls\") pod \"maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc\" (UID: \"d6203004-160a-49e6-bea5-6fe9e8dcdfea\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc"
Sep 11 17:31:54.895152 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:54.894979    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6203004-160a-49e6-bea5-6fe9e8dcdfea-metrics-certs\") pod \"maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc\" (UID: \"d6203004-160a-49e6-bea5-6fe9e8dcdfea\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc"
Sep 11 17:31:54.895152 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:54.895085    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42jbt\" (UniqueName: \"kubernetes.io/projected/d6203004-160a-49e6-bea5-6fe9e8dcdfea-kube-api-access-42jbt\") pod \"maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc\" (UID: \"d6203004-160a-49e6-bea5-6fe9e8dcdfea\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc"
Sep 11 17:31:54.895152 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:54.895115    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6203004-160a-49e6-bea5-6fe9e8dcdfea-openshift-service-ca\") pod \"maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc\" (UID: \"d6203004-160a-49e6-bea5-6fe9e8dcdfea\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc"
Sep 11 17:31:54.996045 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:54.995987    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42jbt\" (UniqueName: \"kubernetes.io/projected/d6203004-160a-49e6-bea5-6fe9e8dcdfea-kube-api-access-42jbt\") pod \"maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc\" (UID: \"d6203004-160a-49e6-bea5-6fe9e8dcdfea\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc"
Sep 11 17:31:54.996253 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:54.996059    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6203004-160a-49e6-bea5-6fe9e8dcdfea-openshift-service-ca\") pod \"maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc\" (UID: \"d6203004-160a-49e6-bea5-6fe9e8dcdfea\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc"
Sep 11 17:31:54.996253 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:54.996115    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6203004-160a-49e6-bea5-6fe9e8dcdfea-maas-api-tls\") pod \"maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc\" (UID: \"d6203004-160a-49e6-bea5-6fe9e8dcdfea\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc"
Sep 11 17:31:54.996253 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:54.996150    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6203004-160a-49e6-bea5-6fe9e8dcdfea-metrics-certs\") pod \"maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc\" (UID: \"d6203004-160a-49e6-bea5-6fe9e8dcdfea\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc"
Sep 11 17:31:54.996826 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:54.996798    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6203004-160a-49e6-bea5-6fe9e8dcdfea-openshift-service-ca\") pod \"maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc\" (UID: \"d6203004-160a-49e6-bea5-6fe9e8dcdfea\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc"
Sep 11 17:31:54.998616 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:54.998596    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6203004-160a-49e6-bea5-6fe9e8dcdfea-maas-api-tls\") pod \"maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc\" (UID: \"d6203004-160a-49e6-bea5-6fe9e8dcdfea\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc"
Sep 11 17:31:54.998692 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:54.998640    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6203004-160a-49e6-bea5-6fe9e8dcdfea-metrics-certs\") pod \"maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc\" (UID: \"d6203004-160a-49e6-bea5-6fe9e8dcdfea\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc"
Sep 11 17:31:55.003643 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:55.003610    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42jbt\" (UniqueName: \"kubernetes.io/projected/d6203004-160a-49e6-bea5-6fe9e8dcdfea-kube-api-access-42jbt\") pod \"maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc\" (UID: \"d6203004-160a-49e6-bea5-6fe9e8dcdfea\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc"
Sep 11 17:31:55.122856 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:55.122817    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc"
Sep 11 17:31:55.394665 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:55.394619    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc"]
Sep 11 17:31:55.397542 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:31:55.397500    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6203004_160a_49e6_bea5_6fe9e8dcdfea.slice/crio-c6ec515a979f5356640214bd7ee967304a0c6f78cdf535ab2071ea1f6757aa83 WatchSource:0}: Error finding container c6ec515a979f5356640214bd7ee967304a0c6f78cdf535ab2071ea1f6757aa83: Status 404 returned error can't find the container with id c6ec515a979f5356640214bd7ee967304a0c6f78cdf535ab2071ea1f6757aa83
Sep 11 17:31:55.499328 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:55.499292    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d19de295-55ccf8c8cd-4nk4h"]
Sep 11 17:31:55.503537 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:55.503515    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d19de295-55ccf8c8cd-4nk4h"
Sep 11 17:31:55.506074 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:55.506049    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-d19de295-dockercfg-rd722\""
Sep 11 17:31:55.506074 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:55.506067    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-d19de295\""
Sep 11 17:31:55.513835 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:55.513811    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d19de295-55ccf8c8cd-4nk4h"]
Sep 11 17:31:55.601243 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:55.601198    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b441ceab-1e56-4d03-bf17-744f65630a63-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d19de295-55ccf8c8cd-4nk4h\" (UID: \"b441ceab-1e56-4d03-bf17-744f65630a63\") " pod="openshift-ingress/payload-processing-e2e-migrate-d19de295-55ccf8c8cd-4nk4h"
Sep 11 17:31:55.601439 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:55.601378    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7kr7\" (UniqueName: \"kubernetes.io/projected/b441ceab-1e56-4d03-bf17-744f65630a63-kube-api-access-p7kr7\") pod \"payload-processing-e2e-migrate-d19de295-55ccf8c8cd-4nk4h\" (UID: \"b441ceab-1e56-4d03-bf17-744f65630a63\") " pod="openshift-ingress/payload-processing-e2e-migrate-d19de295-55ccf8c8cd-4nk4h"
Sep 11 17:31:55.704514 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:55.704407    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7kr7\" (UniqueName: \"kubernetes.io/projected/b441ceab-1e56-4d03-bf17-744f65630a63-kube-api-access-p7kr7\") pod \"payload-processing-e2e-migrate-d19de295-55ccf8c8cd-4nk4h\" (UID: \"b441ceab-1e56-4d03-bf17-744f65630a63\") " pod="openshift-ingress/payload-processing-e2e-migrate-d19de295-55ccf8c8cd-4nk4h"
Sep 11 17:31:55.704746 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:55.704534    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b441ceab-1e56-4d03-bf17-744f65630a63-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d19de295-55ccf8c8cd-4nk4h\" (UID: \"b441ceab-1e56-4d03-bf17-744f65630a63\") " pod="openshift-ingress/payload-processing-e2e-migrate-d19de295-55ccf8c8cd-4nk4h"
Sep 11 17:31:55.706014 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:55.705963    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b441ceab-1e56-4d03-bf17-744f65630a63-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d19de295-55ccf8c8cd-4nk4h\" (UID: \"b441ceab-1e56-4d03-bf17-744f65630a63\") " pod="openshift-ingress/payload-processing-e2e-migrate-d19de295-55ccf8c8cd-4nk4h"
Sep 11 17:31:55.715685 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:55.715661    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7kr7\" (UniqueName: \"kubernetes.io/projected/b441ceab-1e56-4d03-bf17-744f65630a63-kube-api-access-p7kr7\") pod \"payload-processing-e2e-migrate-d19de295-55ccf8c8cd-4nk4h\" (UID: \"b441ceab-1e56-4d03-bf17-744f65630a63\") " pod="openshift-ingress/payload-processing-e2e-migrate-d19de295-55ccf8c8cd-4nk4h"
Sep 11 17:31:55.818406 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:55.818377    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d19de295-55ccf8c8cd-4nk4h"
Sep 11 17:31:56.094691 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:56.094653    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d19de295-55ccf8c8cd-4nk4h"]
Sep 11 17:31:56.097518 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:31:56.097491    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb441ceab_1e56_4d03_bf17_744f65630a63.slice/crio-6c7fd28b8d349250ee33dc1c04472b4dcc0fc5e996d26f444e6c0274b38d92e3 WatchSource:0}: Error finding container 6c7fd28b8d349250ee33dc1c04472b4dcc0fc5e996d26f444e6c0274b38d92e3: Status 404 returned error can't find the container with id 6c7fd28b8d349250ee33dc1c04472b4dcc0fc5e996d26f444e6c0274b38d92e3
Sep 11 17:31:56.405512 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:56.405422    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d19de295-55ccf8c8cd-4nk4h" event={"ID":"b441ceab-1e56-4d03-bf17-744f65630a63","Type":"ContainerStarted","Data":"b315eff882fe031cf7d0757ced5c3bbcdeb37be254129aff0d633e28ada7a308"}
Sep 11 17:31:56.405512 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:56.405481    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d19de295-55ccf8c8cd-4nk4h" event={"ID":"b441ceab-1e56-4d03-bf17-744f65630a63","Type":"ContainerStarted","Data":"6c7fd28b8d349250ee33dc1c04472b4dcc0fc5e996d26f444e6c0274b38d92e3"}
Sep 11 17:31:56.405748 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:56.405539    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-d19de295-55ccf8c8cd-4nk4h"
Sep 11 17:31:56.407144 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:56.407121    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc" event={"ID":"d6203004-160a-49e6-bea5-6fe9e8dcdfea","Type":"ContainerStarted","Data":"2b59c8af5e464ed9e4df19be369a479669934ae3e66b013a0095fcfb4f59c6b3"}
Sep 11 17:31:56.407277 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:56.407166    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc" event={"ID":"d6203004-160a-49e6-bea5-6fe9e8dcdfea","Type":"ContainerStarted","Data":"c6ec515a979f5356640214bd7ee967304a0c6f78cdf535ab2071ea1f6757aa83"}
Sep 11 17:31:56.407277 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:56.407200    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc"
Sep 11 17:31:56.435949 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:56.435904    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-d19de295-55ccf8c8cd-4nk4h" podStartSLOduration=1.435891419 podStartE2EDuration="1.435891419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:31:56.43387972 +0000 UTC m=+796.827797175" watchObservedRunningTime="2026-09-11 17:31:56.435891419 +0000 UTC m=+796.829808918"
Sep 11 17:31:56.475188 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:31:56.475140    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc" podStartSLOduration=2.034992992 podStartE2EDuration="2.475125762s" totalImagesPullingTime="440.13277ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:31:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:31:56.473146052 +0000 UTC m=+796.867063507" watchObservedRunningTime="2026-09-11 17:31:56.475125762 +0000 UTC m=+796.869043218"
Sep 11 17:32:02.418361 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:02.418272    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc"
Sep 11 17:32:07.414931 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:07.414902    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-d19de295-55ccf8c8cd-4nk4h"
Sep 11 17:32:24.192014 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.191964    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc"]
Sep 11 17:32:24.192767 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.192715    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc" podUID="d6203004-160a-49e6-bea5-6fe9e8dcdfea" containerName="maas-api" containerID="cri-o://2b59c8af5e464ed9e4df19be369a479669934ae3e66b013a0095fcfb4f59c6b3" gracePeriod=30
Sep 11 17:32:24.251588 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.251559    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d19de295-55ccf8c8cd-4nk4h"]
Sep 11 17:32:24.251843 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.251800    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-d19de295-55ccf8c8cd-4nk4h" podUID="b441ceab-1e56-4d03-bf17-744f65630a63" containerName="payload-processing" containerID="cri-o://b315eff882fe031cf7d0757ced5c3bbcdeb37be254129aff0d633e28ada7a308" gracePeriod=30
Sep 11 17:32:24.498066 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.497636    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc"
Sep 11 17:32:24.540115 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.539847    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d19de295-55ccf8c8cd-4nk4h"
Sep 11 17:32:24.546302 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.546257    2562 generic.go:356] "Generic (PLEG): container finished" podID="b441ceab-1e56-4d03-bf17-744f65630a63" containerID="b315eff882fe031cf7d0757ced5c3bbcdeb37be254129aff0d633e28ada7a308" exitCode=0
Sep 11 17:32:24.546537 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.546409    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d19de295-55ccf8c8cd-4nk4h"
Sep 11 17:32:24.546537 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.546531    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d19de295-55ccf8c8cd-4nk4h" event={"ID":"b441ceab-1e56-4d03-bf17-744f65630a63","Type":"ContainerDied","Data":"b315eff882fe031cf7d0757ced5c3bbcdeb37be254129aff0d633e28ada7a308"}
Sep 11 17:32:24.546697 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.546567    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d19de295-55ccf8c8cd-4nk4h" event={"ID":"b441ceab-1e56-4d03-bf17-744f65630a63","Type":"ContainerDied","Data":"6c7fd28b8d349250ee33dc1c04472b4dcc0fc5e996d26f444e6c0274b38d92e3"}
Sep 11 17:32:24.546697 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.546594    2562 scope.go:117] "RemoveContainer" containerID="b315eff882fe031cf7d0757ced5c3bbcdeb37be254129aff0d633e28ada7a308"
Sep 11 17:32:24.552039 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.551896    2562 generic.go:356] "Generic (PLEG): container finished" podID="d6203004-160a-49e6-bea5-6fe9e8dcdfea" containerID="2b59c8af5e464ed9e4df19be369a479669934ae3e66b013a0095fcfb4f59c6b3" exitCode=0
Sep 11 17:32:24.552039 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.551973    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc" event={"ID":"d6203004-160a-49e6-bea5-6fe9e8dcdfea","Type":"ContainerDied","Data":"2b59c8af5e464ed9e4df19be369a479669934ae3e66b013a0095fcfb4f59c6b3"}
Sep 11 17:32:24.552203 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.552047    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc" event={"ID":"d6203004-160a-49e6-bea5-6fe9e8dcdfea","Type":"ContainerDied","Data":"c6ec515a979f5356640214bd7ee967304a0c6f78cdf535ab2071ea1f6757aa83"}
Sep 11 17:32:24.552203 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.552115    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc"
Sep 11 17:32:24.559115 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.559092    2562 scope.go:117] "RemoveContainer" containerID="b315eff882fe031cf7d0757ced5c3bbcdeb37be254129aff0d633e28ada7a308"
Sep 11 17:32:24.559516 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:32:24.559470    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b315eff882fe031cf7d0757ced5c3bbcdeb37be254129aff0d633e28ada7a308\": container with ID starting with b315eff882fe031cf7d0757ced5c3bbcdeb37be254129aff0d633e28ada7a308 not found: ID does not exist" containerID="b315eff882fe031cf7d0757ced5c3bbcdeb37be254129aff0d633e28ada7a308"
Sep 11 17:32:24.559631 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.559512    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b315eff882fe031cf7d0757ced5c3bbcdeb37be254129aff0d633e28ada7a308"} err="failed to get container status \"b315eff882fe031cf7d0757ced5c3bbcdeb37be254129aff0d633e28ada7a308\": rpc error: code = NotFound desc = could not find container \"b315eff882fe031cf7d0757ced5c3bbcdeb37be254129aff0d633e28ada7a308\": container with ID starting with b315eff882fe031cf7d0757ced5c3bbcdeb37be254129aff0d633e28ada7a308 not found: ID does not exist"
Sep 11 17:32:24.559631 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.559537    2562 scope.go:117] "RemoveContainer" containerID="2b59c8af5e464ed9e4df19be369a479669934ae3e66b013a0095fcfb4f59c6b3"
Sep 11 17:32:24.574907 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.573798    2562 scope.go:117] "RemoveContainer" containerID="2b59c8af5e464ed9e4df19be369a479669934ae3e66b013a0095fcfb4f59c6b3"
Sep 11 17:32:24.574907 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:32:24.574576    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b59c8af5e464ed9e4df19be369a479669934ae3e66b013a0095fcfb4f59c6b3\": container with ID starting with 2b59c8af5e464ed9e4df19be369a479669934ae3e66b013a0095fcfb4f59c6b3 not found: ID does not exist" containerID="2b59c8af5e464ed9e4df19be369a479669934ae3e66b013a0095fcfb4f59c6b3"
Sep 11 17:32:24.574907 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.574609    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b59c8af5e464ed9e4df19be369a479669934ae3e66b013a0095fcfb4f59c6b3"} err="failed to get container status \"2b59c8af5e464ed9e4df19be369a479669934ae3e66b013a0095fcfb4f59c6b3\": rpc error: code = NotFound desc = could not find container \"2b59c8af5e464ed9e4df19be369a479669934ae3e66b013a0095fcfb4f59c6b3\": container with ID starting with 2b59c8af5e464ed9e4df19be369a479669934ae3e66b013a0095fcfb4f59c6b3 not found: ID does not exist"
Sep 11 17:32:24.685303 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.684027    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42jbt\" (UniqueName: \"kubernetes.io/projected/d6203004-160a-49e6-bea5-6fe9e8dcdfea-kube-api-access-42jbt\") pod \"d6203004-160a-49e6-bea5-6fe9e8dcdfea\" (UID: \"d6203004-160a-49e6-bea5-6fe9e8dcdfea\") "
Sep 11 17:32:24.685303 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.684096    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6203004-160a-49e6-bea5-6fe9e8dcdfea-metrics-certs\") pod \"d6203004-160a-49e6-bea5-6fe9e8dcdfea\" (UID: \"d6203004-160a-49e6-bea5-6fe9e8dcdfea\") "
Sep 11 17:32:24.685303 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.684159    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6203004-160a-49e6-bea5-6fe9e8dcdfea-openshift-service-ca\") pod \"d6203004-160a-49e6-bea5-6fe9e8dcdfea\" (UID: \"d6203004-160a-49e6-bea5-6fe9e8dcdfea\") "
Sep 11 17:32:24.685303 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.684237    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b441ceab-1e56-4d03-bf17-744f65630a63-plugins-config-volume\") pod \"b441ceab-1e56-4d03-bf17-744f65630a63\" (UID: \"b441ceab-1e56-4d03-bf17-744f65630a63\") "
Sep 11 17:32:24.685303 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.684297    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7kr7\" (UniqueName: \"kubernetes.io/projected/b441ceab-1e56-4d03-bf17-744f65630a63-kube-api-access-p7kr7\") pod \"b441ceab-1e56-4d03-bf17-744f65630a63\" (UID: \"b441ceab-1e56-4d03-bf17-744f65630a63\") "
Sep 11 17:32:24.685303 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.684326    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6203004-160a-49e6-bea5-6fe9e8dcdfea-maas-api-tls\") pod \"d6203004-160a-49e6-bea5-6fe9e8dcdfea\" (UID: \"d6203004-160a-49e6-bea5-6fe9e8dcdfea\") "
Sep 11 17:32:24.685757 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.685396    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6203004-160a-49e6-bea5-6fe9e8dcdfea-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d6203004-160a-49e6-bea5-6fe9e8dcdfea" (UID: "d6203004-160a-49e6-bea5-6fe9e8dcdfea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:32:24.686145 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.686107    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b441ceab-1e56-4d03-bf17-744f65630a63-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b441ceab-1e56-4d03-bf17-744f65630a63" (UID: "b441ceab-1e56-4d03-bf17-744f65630a63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:32:24.688057 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.688018    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6203004-160a-49e6-bea5-6fe9e8dcdfea-kube-api-access-42jbt" (OuterVolumeSpecName: "kube-api-access-42jbt") pod "d6203004-160a-49e6-bea5-6fe9e8dcdfea" (UID: "d6203004-160a-49e6-bea5-6fe9e8dcdfea"). InnerVolumeSpecName "kube-api-access-42jbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:32:24.688394 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.688358    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b441ceab-1e56-4d03-bf17-744f65630a63-kube-api-access-p7kr7" (OuterVolumeSpecName: "kube-api-access-p7kr7") pod "b441ceab-1e56-4d03-bf17-744f65630a63" (UID: "b441ceab-1e56-4d03-bf17-744f65630a63"). InnerVolumeSpecName "kube-api-access-p7kr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:32:24.688394 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.688373    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6203004-160a-49e6-bea5-6fe9e8dcdfea-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d6203004-160a-49e6-bea5-6fe9e8dcdfea" (UID: "d6203004-160a-49e6-bea5-6fe9e8dcdfea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:32:24.688561 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.688440    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6203004-160a-49e6-bea5-6fe9e8dcdfea-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d6203004-160a-49e6-bea5-6fe9e8dcdfea" (UID: "d6203004-160a-49e6-bea5-6fe9e8dcdfea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:32:24.786052 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.785970    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7kr7\" (UniqueName: \"kubernetes.io/projected/b441ceab-1e56-4d03-bf17-744f65630a63-kube-api-access-p7kr7\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:32:24.786052 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.786021    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6203004-160a-49e6-bea5-6fe9e8dcdfea-maas-api-tls\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:32:24.786052 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.786037    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42jbt\" (UniqueName: \"kubernetes.io/projected/d6203004-160a-49e6-bea5-6fe9e8dcdfea-kube-api-access-42jbt\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:32:24.786052 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.786046    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6203004-160a-49e6-bea5-6fe9e8dcdfea-metrics-certs\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:32:24.786052 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.786055    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6203004-160a-49e6-bea5-6fe9e8dcdfea-openshift-service-ca\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:32:24.786365 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.786065    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b441ceab-1e56-4d03-bf17-744f65630a63-plugins-config-volume\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:32:24.867776 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.867747    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d19de295-55ccf8c8cd-4nk4h"]
Sep 11 17:32:24.871935 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.871911    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d19de295-55ccf8c8cd-4nk4h"]
Sep 11 17:32:24.881419 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.881394    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc"]
Sep 11 17:32:24.885609 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:24.885585    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d19de295-867fd875c6-6wbpc"]
Sep 11 17:32:25.298392 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.298354    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"]
Sep 11 17:32:25.299088 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.299063    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6203004-160a-49e6-bea5-6fe9e8dcdfea" containerName="maas-api"
Sep 11 17:32:25.299399 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.299380    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6203004-160a-49e6-bea5-6fe9e8dcdfea" containerName="maas-api"
Sep 11 17:32:25.299575 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.299555    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b441ceab-1e56-4d03-bf17-744f65630a63" containerName="payload-processing"
Sep 11 17:32:25.299575 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.299575    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b441ceab-1e56-4d03-bf17-744f65630a63" containerName="payload-processing"
Sep 11 17:32:25.299751 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.299714    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6203004-160a-49e6-bea5-6fe9e8dcdfea" containerName="maas-api"
Sep 11 17:32:25.299751 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.299733    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b441ceab-1e56-4d03-bf17-744f65630a63" containerName="payload-processing"
Sep 11 17:32:25.304414 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.304380    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"
Sep 11 17:32:25.307954 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.307926    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0ad997be-openshift-default-dockercfg-r99pc\""
Sep 11 17:32:25.319219 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.319188    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"]
Sep 11 17:32:25.492877 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.492793    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec7faa6-79e1-4e52-b823-55407f1a32d6-credential-socket\") pod \"e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m\" (UID: \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\") " pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"
Sep 11 17:32:25.492877 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.492838    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ec7faa6-79e1-4e52-b823-55407f1a32d6-istiod-ca-cert\") pod \"e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m\" (UID: \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\") " pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"
Sep 11 17:32:25.493082 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.492942    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec7faa6-79e1-4e52-b823-55407f1a32d6-workload-socket\") pod \"e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m\" (UID: \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\") " pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"
Sep 11 17:32:25.493082 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.492973    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ec7faa6-79e1-4e52-b823-55407f1a32d6-istio-envoy\") pod \"e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m\" (UID: \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\") " pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"
Sep 11 17:32:25.493082 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.493036    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ec7faa6-79e1-4e52-b823-55407f1a32d6-istio-data\") pod \"e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m\" (UID: \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\") " pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"
Sep 11 17:32:25.493183 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.493089    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6zbs\" (UniqueName: \"kubernetes.io/projected/2ec7faa6-79e1-4e52-b823-55407f1a32d6-kube-api-access-v6zbs\") pod \"e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m\" (UID: \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\") " pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"
Sep 11 17:32:25.493183 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.493159    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ec7faa6-79e1-4e52-b823-55407f1a32d6-workload-certs\") pod \"e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m\" (UID: \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\") " pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"
Sep 11 17:32:25.493245 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.493184    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ec7faa6-79e1-4e52-b823-55407f1a32d6-istio-podinfo\") pod \"e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m\" (UID: \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\") " pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"
Sep 11 17:32:25.493245 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.493206    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ec7faa6-79e1-4e52-b823-55407f1a32d6-istio-token\") pod \"e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m\" (UID: \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\") " pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"
Sep 11 17:32:25.594179 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.594143    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ec7faa6-79e1-4e52-b823-55407f1a32d6-istio-data\") pod \"e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m\" (UID: \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\") " pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"
Sep 11 17:32:25.594179 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.594178    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6zbs\" (UniqueName: \"kubernetes.io/projected/2ec7faa6-79e1-4e52-b823-55407f1a32d6-kube-api-access-v6zbs\") pod \"e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m\" (UID: \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\") " pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"
Sep 11 17:32:25.594445 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.594206    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ec7faa6-79e1-4e52-b823-55407f1a32d6-workload-certs\") pod \"e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m\" (UID: \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\") " pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"
Sep 11 17:32:25.594445 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.594231    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ec7faa6-79e1-4e52-b823-55407f1a32d6-istio-podinfo\") pod \"e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m\" (UID: \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\") " pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"
Sep 11 17:32:25.594445 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.594277    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ec7faa6-79e1-4e52-b823-55407f1a32d6-istio-token\") pod \"e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m\" (UID: \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\") " pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"
Sep 11 17:32:25.594445 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.594342    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec7faa6-79e1-4e52-b823-55407f1a32d6-credential-socket\") pod \"e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m\" (UID: \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\") " pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"
Sep 11 17:32:25.594445 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.594366    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ec7faa6-79e1-4e52-b823-55407f1a32d6-istiod-ca-cert\") pod \"e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m\" (UID: \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\") " pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"
Sep 11 17:32:25.594445 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.594431    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec7faa6-79e1-4e52-b823-55407f1a32d6-workload-socket\") pod \"e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m\" (UID: \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\") " pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"
Sep 11 17:32:25.594746 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.594464    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ec7faa6-79e1-4e52-b823-55407f1a32d6-istio-envoy\") pod \"e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m\" (UID: \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\") " pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"
Sep 11 17:32:25.594746 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.594675    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ec7faa6-79e1-4e52-b823-55407f1a32d6-istio-data\") pod \"e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m\" (UID: \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\") " pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"
Sep 11 17:32:25.594746 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.594713    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec7faa6-79e1-4e52-b823-55407f1a32d6-credential-socket\") pod \"e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m\" (UID: \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\") " pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"
Sep 11 17:32:25.594841 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.594771    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ec7faa6-79e1-4e52-b823-55407f1a32d6-workload-certs\") pod \"e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m\" (UID: \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\") " pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"
Sep 11 17:32:25.595059 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.595028    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec7faa6-79e1-4e52-b823-55407f1a32d6-workload-socket\") pod \"e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m\" (UID: \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\") " pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"
Sep 11 17:32:25.595192 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.595175    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ec7faa6-79e1-4e52-b823-55407f1a32d6-istiod-ca-cert\") pod \"e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m\" (UID: \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\") " pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"
Sep 11 17:32:25.596660 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.596636    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ec7faa6-79e1-4e52-b823-55407f1a32d6-istio-envoy\") pod \"e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m\" (UID: \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\") " pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"
Sep 11 17:32:25.596754 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.596737    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ec7faa6-79e1-4e52-b823-55407f1a32d6-istio-podinfo\") pod \"e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m\" (UID: \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\") " pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"
Sep 11 17:32:25.602035 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.601970    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ec7faa6-79e1-4e52-b823-55407f1a32d6-istio-token\") pod \"e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m\" (UID: \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\") " pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"
Sep 11 17:32:25.602188 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.602167    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6zbs\" (UniqueName: \"kubernetes.io/projected/2ec7faa6-79e1-4e52-b823-55407f1a32d6-kube-api-access-v6zbs\") pod \"e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m\" (UID: \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\") " pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"
Sep 11 17:32:25.621005 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.620974    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"
Sep 11 17:32:25.895843 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.895822    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"]
Sep 11 17:32:25.897859 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:32:25.897828    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ec7faa6_79e1_4e52_b823_55407f1a32d6.slice/crio-1d6961adc141c898cf4571d7779b58504af563a1227f6f84bbc59d55ece1795a WatchSource:0}: Error finding container 1d6961adc141c898cf4571d7779b58504af563a1227f6f84bbc59d55ece1795a: Status 404 returned error can't find the container with id 1d6961adc141c898cf4571d7779b58504af563a1227f6f84bbc59d55ece1795a
Sep 11 17:32:25.900268 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.900234    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:32:25.900350 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.900301    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:32:25.900350 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:25.900331    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:32:26.127358 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:26.127318    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b441ceab-1e56-4d03-bf17-744f65630a63" path="/var/lib/kubelet/pods/b441ceab-1e56-4d03-bf17-744f65630a63/volumes"
Sep 11 17:32:26.127943 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:26.127924    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6203004-160a-49e6-bea5-6fe9e8dcdfea" path="/var/lib/kubelet/pods/d6203004-160a-49e6-bea5-6fe9e8dcdfea/volumes"
Sep 11 17:32:26.565439 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:26.565406    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m" event={"ID":"2ec7faa6-79e1-4e52-b823-55407f1a32d6","Type":"ContainerStarted","Data":"b0293b2e5d0264da9b452860716589ebafc0158941a034962149e554abb32e62"}
Sep 11 17:32:26.565829 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:26.565456    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m" event={"ID":"2ec7faa6-79e1-4e52-b823-55407f1a32d6","Type":"ContainerStarted","Data":"1d6961adc141c898cf4571d7779b58504af563a1227f6f84bbc59d55ece1795a"}
Sep 11 17:32:26.583346 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:26.583297    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m" podStartSLOduration=1.583279544 podStartE2EDuration="1.583279544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:32:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:32:26.583200082 +0000 UTC m=+826.977117531" watchObservedRunningTime="2026-09-11 17:32:26.583279544 +0000 UTC m=+826.977197003"
Sep 11 17:32:26.621108 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:26.621077    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"
Sep 11 17:32:26.626607 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:26.626582    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"
Sep 11 17:32:27.570817 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:27.570778    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"
Sep 11 17:32:27.571643 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:27.571627    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"
Sep 11 17:32:30.101706 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:30.101674    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0ad997be-79c4d46cd-2lb6n"]
Sep 11 17:32:30.105261 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:30.105244    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ad997be-79c4d46cd-2lb6n"
Sep 11 17:32:30.107367 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:30.107345    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0ad997be\""
Sep 11 17:32:30.116333 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:30.116308    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0ad997be-79c4d46cd-2lb6n"]
Sep 11 17:32:30.141663 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:30.141624    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8x6g\" (UniqueName: \"kubernetes.io/projected/2cd23851-76f5-4715-9c7a-f95dc7b075d7-kube-api-access-n8x6g\") pod \"payload-pre-processing-e2e-ait-0ad997be-79c4d46cd-2lb6n\" (UID: \"2cd23851-76f5-4715-9c7a-f95dc7b075d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0ad997be-79c4d46cd-2lb6n"
Sep 11 17:32:30.141864 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:30.141760    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cd23851-76f5-4715-9c7a-f95dc7b075d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0ad997be-79c4d46cd-2lb6n\" (UID: \"2cd23851-76f5-4715-9c7a-f95dc7b075d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0ad997be-79c4d46cd-2lb6n"
Sep 11 17:32:30.144028 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:30.143984    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0ad997be-75796d85d9-httpc"]
Sep 11 17:32:30.146621 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:30.146602    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0ad997be-75796d85d9-httpc"
Sep 11 17:32:30.149048 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:30.149026    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0ad997be-dockercfg-vrqdc\""
Sep 11 17:32:30.158659 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:30.158635    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0ad997be-75796d85d9-httpc"]
Sep 11 17:32:30.243242 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:30.242813    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54vgj\" (UniqueName: \"kubernetes.io/projected/1000856f-12e0-4f2e-b3a7-1d39dc1d3b41-kube-api-access-54vgj\") pod \"payload-processing-e2e-ait-0ad997be-75796d85d9-httpc\" (UID: \"1000856f-12e0-4f2e-b3a7-1d39dc1d3b41\") " pod="openshift-ingress/payload-processing-e2e-ait-0ad997be-75796d85d9-httpc"
Sep 11 17:32:30.243242 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:30.242959    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8x6g\" (UniqueName: \"kubernetes.io/projected/2cd23851-76f5-4715-9c7a-f95dc7b075d7-kube-api-access-n8x6g\") pod \"payload-pre-processing-e2e-ait-0ad997be-79c4d46cd-2lb6n\" (UID: \"2cd23851-76f5-4715-9c7a-f95dc7b075d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0ad997be-79c4d46cd-2lb6n"
Sep 11 17:32:30.243242 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:30.243039    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1000856f-12e0-4f2e-b3a7-1d39dc1d3b41-plugins-config-volume\") pod \"payload-processing-e2e-ait-0ad997be-75796d85d9-httpc\" (UID: \"1000856f-12e0-4f2e-b3a7-1d39dc1d3b41\") " pod="openshift-ingress/payload-processing-e2e-ait-0ad997be-75796d85d9-httpc"
Sep 11 17:32:30.243540 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:30.243113    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cd23851-76f5-4715-9c7a-f95dc7b075d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0ad997be-79c4d46cd-2lb6n\" (UID: \"2cd23851-76f5-4715-9c7a-f95dc7b075d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0ad997be-79c4d46cd-2lb6n"
Sep 11 17:32:30.244178 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:30.244153    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cd23851-76f5-4715-9c7a-f95dc7b075d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0ad997be-79c4d46cd-2lb6n\" (UID: \"2cd23851-76f5-4715-9c7a-f95dc7b075d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0ad997be-79c4d46cd-2lb6n"
Sep 11 17:32:30.253036 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:30.252980    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8x6g\" (UniqueName: \"kubernetes.io/projected/2cd23851-76f5-4715-9c7a-f95dc7b075d7-kube-api-access-n8x6g\") pod \"payload-pre-processing-e2e-ait-0ad997be-79c4d46cd-2lb6n\" (UID: \"2cd23851-76f5-4715-9c7a-f95dc7b075d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0ad997be-79c4d46cd-2lb6n"
Sep 11 17:32:30.344905 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:30.344862    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54vgj\" (UniqueName: \"kubernetes.io/projected/1000856f-12e0-4f2e-b3a7-1d39dc1d3b41-kube-api-access-54vgj\") pod \"payload-processing-e2e-ait-0ad997be-75796d85d9-httpc\" (UID: \"1000856f-12e0-4f2e-b3a7-1d39dc1d3b41\") " pod="openshift-ingress/payload-processing-e2e-ait-0ad997be-75796d85d9-httpc"
Sep 11 17:32:30.345115 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:30.344985    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1000856f-12e0-4f2e-b3a7-1d39dc1d3b41-plugins-config-volume\") pod \"payload-processing-e2e-ait-0ad997be-75796d85d9-httpc\" (UID: \"1000856f-12e0-4f2e-b3a7-1d39dc1d3b41\") " pod="openshift-ingress/payload-processing-e2e-ait-0ad997be-75796d85d9-httpc"
Sep 11 17:32:30.345680 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:30.345659    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1000856f-12e0-4f2e-b3a7-1d39dc1d3b41-plugins-config-volume\") pod \"payload-processing-e2e-ait-0ad997be-75796d85d9-httpc\" (UID: \"1000856f-12e0-4f2e-b3a7-1d39dc1d3b41\") " pod="openshift-ingress/payload-processing-e2e-ait-0ad997be-75796d85d9-httpc"
Sep 11 17:32:30.352897 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:30.352843    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54vgj\" (UniqueName: \"kubernetes.io/projected/1000856f-12e0-4f2e-b3a7-1d39dc1d3b41-kube-api-access-54vgj\") pod \"payload-processing-e2e-ait-0ad997be-75796d85d9-httpc\" (UID: \"1000856f-12e0-4f2e-b3a7-1d39dc1d3b41\") " pod="openshift-ingress/payload-processing-e2e-ait-0ad997be-75796d85d9-httpc"
Sep 11 17:32:30.417891 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:30.417849    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ad997be-79c4d46cd-2lb6n"
Sep 11 17:32:30.457871 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:30.457840    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0ad997be-75796d85d9-httpc"
Sep 11 17:32:30.695271 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:30.695247    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0ad997be-79c4d46cd-2lb6n"]
Sep 11 17:32:30.697107 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:32:30.697082    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cd23851_76f5_4715_9c7a_f95dc7b075d7.slice/crio-62c1031454c530dbfcc9222d0149cd57790b335dc33652ad4175d4b9940de89b WatchSource:0}: Error finding container 62c1031454c530dbfcc9222d0149cd57790b335dc33652ad4175d4b9940de89b: Status 404 returned error can't find the container with id 62c1031454c530dbfcc9222d0149cd57790b335dc33652ad4175d4b9940de89b
Sep 11 17:32:30.749030 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:30.748982    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0ad997be-75796d85d9-httpc"]
Sep 11 17:32:30.750972 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:32:30.750934    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1000856f_12e0_4f2e_b3a7_1d39dc1d3b41.slice/crio-2336d3700c62ed3f0645645da3df8100db4ba07fc60acb490c71fa41ca9ea429 WatchSource:0}: Error finding container 2336d3700c62ed3f0645645da3df8100db4ba07fc60acb490c71fa41ca9ea429: Status 404 returned error can't find the container with id 2336d3700c62ed3f0645645da3df8100db4ba07fc60acb490c71fa41ca9ea429
Sep 11 17:32:31.592869 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:31.592825    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0ad997be-75796d85d9-httpc" event={"ID":"1000856f-12e0-4f2e-b3a7-1d39dc1d3b41","Type":"ContainerStarted","Data":"6f8322c94854eb54f84bdd013009d4a21b0981524dccbe98a83629a9a8ae7ef8"}
Sep 11 17:32:31.593369 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:31.592892    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0ad997be-75796d85d9-httpc" event={"ID":"1000856f-12e0-4f2e-b3a7-1d39dc1d3b41","Type":"ContainerStarted","Data":"2336d3700c62ed3f0645645da3df8100db4ba07fc60acb490c71fa41ca9ea429"}
Sep 11 17:32:31.593369 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:31.592922    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0ad997be-75796d85d9-httpc"
Sep 11 17:32:31.594651 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:31.594625    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ad997be-79c4d46cd-2lb6n" event={"ID":"2cd23851-76f5-4715-9c7a-f95dc7b075d7","Type":"ContainerStarted","Data":"699bd95eaf784219324a6c8884230344d296b04f87aef6cb13cfdc1d63b1a9a3"}
Sep 11 17:32:31.594720 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:31.594668    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ad997be-79c4d46cd-2lb6n" event={"ID":"2cd23851-76f5-4715-9c7a-f95dc7b075d7","Type":"ContainerStarted","Data":"62c1031454c530dbfcc9222d0149cd57790b335dc33652ad4175d4b9940de89b"}
Sep 11 17:32:31.594781 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:31.594715    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ad997be-79c4d46cd-2lb6n"
Sep 11 17:32:31.608451 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:31.608397    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0ad997be-75796d85d9-httpc" podStartSLOduration=1.6083777339999998 podStartE2EDuration="1.608377734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:32:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:32:31.607756053 +0000 UTC m=+832.001673508" watchObservedRunningTime="2026-09-11 17:32:31.608377734 +0000 UTC m=+832.002295189"
Sep 11 17:32:31.622914 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:31.622855    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ad997be-79c4d46cd-2lb6n" podStartSLOduration=1.622836255 podStartE2EDuration="1.622836255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:32:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:32:31.622696807 +0000 UTC m=+832.016614262" watchObservedRunningTime="2026-09-11 17:32:31.622836255 +0000 UTC m=+832.016753713"
Sep 11 17:32:42.603166 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:42.603130    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ad997be-79c4d46cd-2lb6n"
Sep 11 17:32:42.603584 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:42.603200    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0ad997be-75796d85d9-httpc"
Sep 11 17:32:51.714959 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:51.714925    2562 generic.go:356] "Generic (PLEG): container finished" podID="2ec7faa6-79e1-4e52-b823-55407f1a32d6" containerID="b0293b2e5d0264da9b452860716589ebafc0158941a034962149e554abb32e62" exitCode=0
Sep 11 17:32:51.715396 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:51.715011    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m" event={"ID":"2ec7faa6-79e1-4e52-b823-55407f1a32d6","Type":"ContainerDied","Data":"b0293b2e5d0264da9b452860716589ebafc0158941a034962149e554abb32e62"}
Sep 11 17:32:51.715447 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:51.715396    2562 scope.go:117] "RemoveContainer" containerID="b0293b2e5d0264da9b452860716589ebafc0158941a034962149e554abb32e62"
Sep 11 17:32:51.716489 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:51.716462    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:32:51.716551 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:51.716529    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:32:51.716612 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:51.716559    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:32:52.722041 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:52.722007    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m" event={"ID":"2ec7faa6-79e1-4e52-b823-55407f1a32d6","Type":"ContainerStarted","Data":"a67a985df51fb892eb4de5f7e3291b6e941630ea0c9fe2b15344d7946ff5f951"}
Sep 11 17:32:53.621567 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:53.621527    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"
Sep 11 17:32:53.622847 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:53.622817    2562 patch_prober.go:28] interesting pod/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 11 17:32:53.623023 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:53.622870    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m" podUID="2ec7faa6-79e1-4e52-b823-55407f1a32d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 11 17:32:54.622085 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:54.622042    2562 patch_prober.go:28] interesting pod/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 11 17:32:54.622464 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:54.622104    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m" podUID="2ec7faa6-79e1-4e52-b823-55407f1a32d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 11 17:32:55.621763 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:55.621728    2562 patch_prober.go:28] interesting pod/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 11 17:32:55.621945 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:55.621801    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m" podUID="2ec7faa6-79e1-4e52-b823-55407f1a32d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 11 17:32:56.621515 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:56.621475    2562 patch_prober.go:28] interesting pod/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 11 17:32:56.622029 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:56.621539    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m" podUID="2ec7faa6-79e1-4e52-b823-55407f1a32d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 11 17:32:57.570892 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:57.570836    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"
Sep 11 17:32:57.621521 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:57.621490    2562 patch_prober.go:28] interesting pod/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 11 17:32:57.621900 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:57.621546    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m" podUID="2ec7faa6-79e1-4e52-b823-55407f1a32d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 11 17:32:58.625546 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:58.625517    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"
Sep 11 17:32:58.626476 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:32:58.626454    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"
Sep 11 17:33:00.600498 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:00.600462    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0ad997be-75796d85d9-httpc"]
Sep 11 17:33:00.600970 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:00.600713    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0ad997be-75796d85d9-httpc" podUID="1000856f-12e0-4f2e-b3a7-1d39dc1d3b41" containerName="payload-processing" containerID="cri-o://6f8322c94854eb54f84bdd013009d4a21b0981524dccbe98a83629a9a8ae7ef8" gracePeriod=30
Sep 11 17:33:00.606589 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:00.606561    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0ad997be-79c4d46cd-2lb6n"]
Sep 11 17:33:00.606826 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:00.606777    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ad997be-79c4d46cd-2lb6n" podUID="2cd23851-76f5-4715-9c7a-f95dc7b075d7" containerName="payload-pre-processing" containerID="cri-o://699bd95eaf784219324a6c8884230344d296b04f87aef6cb13cfdc1d63b1a9a3" gracePeriod=30
Sep 11 17:33:00.766216 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:00.766157    2562 generic.go:356] "Generic (PLEG): container finished" podID="1000856f-12e0-4f2e-b3a7-1d39dc1d3b41" containerID="6f8322c94854eb54f84bdd013009d4a21b0981524dccbe98a83629a9a8ae7ef8" exitCode=0
Sep 11 17:33:00.766370 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:00.766265    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0ad997be-75796d85d9-httpc" event={"ID":"1000856f-12e0-4f2e-b3a7-1d39dc1d3b41","Type":"ContainerDied","Data":"6f8322c94854eb54f84bdd013009d4a21b0981524dccbe98a83629a9a8ae7ef8"}
Sep 11 17:33:00.768439 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:00.768393    2562 generic.go:356] "Generic (PLEG): container finished" podID="2cd23851-76f5-4715-9c7a-f95dc7b075d7" containerID="699bd95eaf784219324a6c8884230344d296b04f87aef6cb13cfdc1d63b1a9a3" exitCode=0
Sep 11 17:33:00.768595 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:00.768499    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ad997be-79c4d46cd-2lb6n" event={"ID":"2cd23851-76f5-4715-9c7a-f95dc7b075d7","Type":"ContainerDied","Data":"699bd95eaf784219324a6c8884230344d296b04f87aef6cb13cfdc1d63b1a9a3"}
Sep 11 17:33:00.873885 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:00.873860    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0ad997be-75796d85d9-httpc"
Sep 11 17:33:00.904251 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:00.904229    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ad997be-79c4d46cd-2lb6n"
Sep 11 17:33:00.946102 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:00.945490    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54vgj\" (UniqueName: \"kubernetes.io/projected/1000856f-12e0-4f2e-b3a7-1d39dc1d3b41-kube-api-access-54vgj\") pod \"1000856f-12e0-4f2e-b3a7-1d39dc1d3b41\" (UID: \"1000856f-12e0-4f2e-b3a7-1d39dc1d3b41\") "
Sep 11 17:33:00.946102 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:00.945558    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1000856f-12e0-4f2e-b3a7-1d39dc1d3b41-plugins-config-volume\") pod \"1000856f-12e0-4f2e-b3a7-1d39dc1d3b41\" (UID: \"1000856f-12e0-4f2e-b3a7-1d39dc1d3b41\") "
Sep 11 17:33:00.947769 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:00.947729    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1000856f-12e0-4f2e-b3a7-1d39dc1d3b41-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1000856f-12e0-4f2e-b3a7-1d39dc1d3b41" (UID: "1000856f-12e0-4f2e-b3a7-1d39dc1d3b41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:33:00.950451 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:00.950405    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1000856f-12e0-4f2e-b3a7-1d39dc1d3b41-kube-api-access-54vgj" (OuterVolumeSpecName: "kube-api-access-54vgj") pod "1000856f-12e0-4f2e-b3a7-1d39dc1d3b41" (UID: "1000856f-12e0-4f2e-b3a7-1d39dc1d3b41"). InnerVolumeSpecName "kube-api-access-54vgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:33:01.047819 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:01.047783    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cd23851-76f5-4715-9c7a-f95dc7b075d7-plugins-config-volume\") pod \"2cd23851-76f5-4715-9c7a-f95dc7b075d7\" (UID: \"2cd23851-76f5-4715-9c7a-f95dc7b075d7\") "
Sep 11 17:33:01.048040 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:01.047863    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8x6g\" (UniqueName: \"kubernetes.io/projected/2cd23851-76f5-4715-9c7a-f95dc7b075d7-kube-api-access-n8x6g\") pod \"2cd23851-76f5-4715-9c7a-f95dc7b075d7\" (UID: \"2cd23851-76f5-4715-9c7a-f95dc7b075d7\") "
Sep 11 17:33:01.048287 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:01.048219    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54vgj\" (UniqueName: \"kubernetes.io/projected/1000856f-12e0-4f2e-b3a7-1d39dc1d3b41-kube-api-access-54vgj\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:33:01.048287 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:01.048246    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1000856f-12e0-4f2e-b3a7-1d39dc1d3b41-plugins-config-volume\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:33:01.048450 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:01.048308    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cd23851-76f5-4715-9c7a-f95dc7b075d7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2cd23851-76f5-4715-9c7a-f95dc7b075d7" (UID: "2cd23851-76f5-4715-9c7a-f95dc7b075d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:33:01.050268 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:01.050235    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cd23851-76f5-4715-9c7a-f95dc7b075d7-kube-api-access-n8x6g" (OuterVolumeSpecName: "kube-api-access-n8x6g") pod "2cd23851-76f5-4715-9c7a-f95dc7b075d7" (UID: "2cd23851-76f5-4715-9c7a-f95dc7b075d7"). InnerVolumeSpecName "kube-api-access-n8x6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:33:01.149515 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:01.149413    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cd23851-76f5-4715-9c7a-f95dc7b075d7-plugins-config-volume\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:33:01.149515 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:01.149447    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8x6g\" (UniqueName: \"kubernetes.io/projected/2cd23851-76f5-4715-9c7a-f95dc7b075d7-kube-api-access-n8x6g\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:33:01.775196 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:01.775165    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0ad997be-75796d85d9-httpc"
Sep 11 17:33:01.775671 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:01.775165    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0ad997be-75796d85d9-httpc" event={"ID":"1000856f-12e0-4f2e-b3a7-1d39dc1d3b41","Type":"ContainerDied","Data":"2336d3700c62ed3f0645645da3df8100db4ba07fc60acb490c71fa41ca9ea429"}
Sep 11 17:33:01.775671 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:01.775295    2562 scope.go:117] "RemoveContainer" containerID="6f8322c94854eb54f84bdd013009d4a21b0981524dccbe98a83629a9a8ae7ef8"
Sep 11 17:33:01.776914 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:01.776704    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ad997be-79c4d46cd-2lb6n"
Sep 11 17:33:01.776914 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:01.776745    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ad997be-79c4d46cd-2lb6n" event={"ID":"2cd23851-76f5-4715-9c7a-f95dc7b075d7","Type":"ContainerDied","Data":"62c1031454c530dbfcc9222d0149cd57790b335dc33652ad4175d4b9940de89b"}
Sep 11 17:33:01.787007 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:01.786971    2562 scope.go:117] "RemoveContainer" containerID="699bd95eaf784219324a6c8884230344d296b04f87aef6cb13cfdc1d63b1a9a3"
Sep 11 17:33:01.800877 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:01.800851    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0ad997be-75796d85d9-httpc"]
Sep 11 17:33:01.808348 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:01.808327    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0ad997be-75796d85d9-httpc"]
Sep 11 17:33:01.820213 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:01.820186    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0ad997be-79c4d46cd-2lb6n"]
Sep 11 17:33:01.824253 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:01.824231    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0ad997be-79c4d46cd-2lb6n"]
Sep 11 17:33:01.919951 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:01.919869    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"]
Sep 11 17:33:01.920140 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:01.920095    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m" podUID="2ec7faa6-79e1-4e52-b823-55407f1a32d6" containerName="istio-proxy" containerID="cri-o://a67a985df51fb892eb4de5f7e3291b6e941630ea0c9fe2b15344d7946ff5f951" gracePeriod=30
Sep 11 17:33:02.125805 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:02.125762    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1000856f-12e0-4f2e-b3a7-1d39dc1d3b41" path="/var/lib/kubelet/pods/1000856f-12e0-4f2e-b3a7-1d39dc1d3b41/volumes"
Sep 11 17:33:02.126193 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:02.126179    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cd23851-76f5-4715-9c7a-f95dc7b075d7" path="/var/lib/kubelet/pods/2cd23851-76f5-4715-9c7a-f95dc7b075d7/volumes"
Sep 11 17:33:07.178573 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.178539    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"
Sep 11 17:33:07.312259 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.312174    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec7faa6-79e1-4e52-b823-55407f1a32d6-credential-socket\") pod \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\" (UID: \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\") "
Sep 11 17:33:07.312259 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.312225    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6zbs\" (UniqueName: \"kubernetes.io/projected/2ec7faa6-79e1-4e52-b823-55407f1a32d6-kube-api-access-v6zbs\") pod \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\" (UID: \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\") "
Sep 11 17:33:07.312460 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.312277    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ec7faa6-79e1-4e52-b823-55407f1a32d6-istio-data\") pod \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\" (UID: \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\") "
Sep 11 17:33:07.312460 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.312301    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec7faa6-79e1-4e52-b823-55407f1a32d6-workload-socket\") pod \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\" (UID: \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\") "
Sep 11 17:33:07.312460 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.312330    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ec7faa6-79e1-4e52-b823-55407f1a32d6-workload-certs\") pod \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\" (UID: \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\") "
Sep 11 17:33:07.312460 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.312356    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ec7faa6-79e1-4e52-b823-55407f1a32d6-istio-envoy\") pod \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\" (UID: \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\") "
Sep 11 17:33:07.312460 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.312409    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ec7faa6-79e1-4e52-b823-55407f1a32d6-istiod-ca-cert\") pod \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\" (UID: \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\") "
Sep 11 17:33:07.312708 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.312471    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ec7faa6-79e1-4e52-b823-55407f1a32d6-istio-podinfo\") pod \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\" (UID: \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\") "
Sep 11 17:33:07.312708 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.312503    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ec7faa6-79e1-4e52-b823-55407f1a32d6-istio-token\") pod \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\" (UID: \"2ec7faa6-79e1-4e52-b823-55407f1a32d6\") "
Sep 11 17:33:07.312708 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.312507    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec7faa6-79e1-4e52-b823-55407f1a32d6-istio-data" (OuterVolumeSpecName: "istio-data") pod "2ec7faa6-79e1-4e52-b823-55407f1a32d6" (UID: "2ec7faa6-79e1-4e52-b823-55407f1a32d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:33:07.312708 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.312521    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec7faa6-79e1-4e52-b823-55407f1a32d6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2ec7faa6-79e1-4e52-b823-55407f1a32d6" (UID: "2ec7faa6-79e1-4e52-b823-55407f1a32d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:33:07.312708 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.312649    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec7faa6-79e1-4e52-b823-55407f1a32d6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2ec7faa6-79e1-4e52-b823-55407f1a32d6" (UID: "2ec7faa6-79e1-4e52-b823-55407f1a32d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:33:07.312708 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.312643    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec7faa6-79e1-4e52-b823-55407f1a32d6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2ec7faa6-79e1-4e52-b823-55407f1a32d6" (UID: "2ec7faa6-79e1-4e52-b823-55407f1a32d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:33:07.313041 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.312886    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec7faa6-79e1-4e52-b823-55407f1a32d6-credential-socket\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:33:07.313041 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.312886    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ec7faa6-79e1-4e52-b823-55407f1a32d6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2ec7faa6-79e1-4e52-b823-55407f1a32d6" (UID: "2ec7faa6-79e1-4e52-b823-55407f1a32d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:33:07.313041 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.312907    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ec7faa6-79e1-4e52-b823-55407f1a32d6-istio-data\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:33:07.313041 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.312921    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec7faa6-79e1-4e52-b823-55407f1a32d6-workload-socket\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:33:07.313041 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.312932    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ec7faa6-79e1-4e52-b823-55407f1a32d6-workload-certs\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:33:07.314912 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.314879    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ec7faa6-79e1-4e52-b823-55407f1a32d6-istio-token" (OuterVolumeSpecName: "istio-token") pod "2ec7faa6-79e1-4e52-b823-55407f1a32d6" (UID: "2ec7faa6-79e1-4e52-b823-55407f1a32d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:33:07.315065 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.314983    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ec7faa6-79e1-4e52-b823-55407f1a32d6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2ec7faa6-79e1-4e52-b823-55407f1a32d6" (UID: "2ec7faa6-79e1-4e52-b823-55407f1a32d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 17:33:07.315122 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.315080    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ec7faa6-79e1-4e52-b823-55407f1a32d6-kube-api-access-v6zbs" (OuterVolumeSpecName: "kube-api-access-v6zbs") pod "2ec7faa6-79e1-4e52-b823-55407f1a32d6" (UID: "2ec7faa6-79e1-4e52-b823-55407f1a32d6"). InnerVolumeSpecName "kube-api-access-v6zbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:33:07.315122 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.315074    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec7faa6-79e1-4e52-b823-55407f1a32d6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2ec7faa6-79e1-4e52-b823-55407f1a32d6" (UID: "2ec7faa6-79e1-4e52-b823-55407f1a32d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:33:07.414401 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.414358    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ec7faa6-79e1-4e52-b823-55407f1a32d6-istio-token\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:33:07.414401 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.414397    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6zbs\" (UniqueName: \"kubernetes.io/projected/2ec7faa6-79e1-4e52-b823-55407f1a32d6-kube-api-access-v6zbs\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:33:07.414401 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.414412    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ec7faa6-79e1-4e52-b823-55407f1a32d6-istio-envoy\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:33:07.414741 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.414425    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ec7faa6-79e1-4e52-b823-55407f1a32d6-istiod-ca-cert\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:33:07.414741 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.414482    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ec7faa6-79e1-4e52-b823-55407f1a32d6-istio-podinfo\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:33:07.809209 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.809177    2562 generic.go:356] "Generic (PLEG): container finished" podID="2ec7faa6-79e1-4e52-b823-55407f1a32d6" containerID="a67a985df51fb892eb4de5f7e3291b6e941630ea0c9fe2b15344d7946ff5f951" exitCode=0
Sep 11 17:33:07.809375 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.809228    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"
Sep 11 17:33:07.809375 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.809229    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m" event={"ID":"2ec7faa6-79e1-4e52-b823-55407f1a32d6","Type":"ContainerDied","Data":"a67a985df51fb892eb4de5f7e3291b6e941630ea0c9fe2b15344d7946ff5f951"}
Sep 11 17:33:07.809375 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.809352    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m" event={"ID":"2ec7faa6-79e1-4e52-b823-55407f1a32d6","Type":"ContainerDied","Data":"1d6961adc141c898cf4571d7779b58504af563a1227f6f84bbc59d55ece1795a"}
Sep 11 17:33:07.809480 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.809383    2562 scope.go:117] "RemoveContainer" containerID="a67a985df51fb892eb4de5f7e3291b6e941630ea0c9fe2b15344d7946ff5f951"
Sep 11 17:33:07.818880 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.818865    2562 scope.go:117] "RemoveContainer" containerID="b0293b2e5d0264da9b452860716589ebafc0158941a034962149e554abb32e62"
Sep 11 17:33:07.826920 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.826903    2562 scope.go:117] "RemoveContainer" containerID="a67a985df51fb892eb4de5f7e3291b6e941630ea0c9fe2b15344d7946ff5f951"
Sep 11 17:33:07.827211 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:33:07.827185    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a67a985df51fb892eb4de5f7e3291b6e941630ea0c9fe2b15344d7946ff5f951\": container with ID starting with a67a985df51fb892eb4de5f7e3291b6e941630ea0c9fe2b15344d7946ff5f951 not found: ID does not exist" containerID="a67a985df51fb892eb4de5f7e3291b6e941630ea0c9fe2b15344d7946ff5f951"
Sep 11 17:33:07.827326 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.827222    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a67a985df51fb892eb4de5f7e3291b6e941630ea0c9fe2b15344d7946ff5f951"} err="failed to get container status \"a67a985df51fb892eb4de5f7e3291b6e941630ea0c9fe2b15344d7946ff5f951\": rpc error: code = NotFound desc = could not find container \"a67a985df51fb892eb4de5f7e3291b6e941630ea0c9fe2b15344d7946ff5f951\": container with ID starting with a67a985df51fb892eb4de5f7e3291b6e941630ea0c9fe2b15344d7946ff5f951 not found: ID does not exist"
Sep 11 17:33:07.827326 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.827250    2562 scope.go:117] "RemoveContainer" containerID="b0293b2e5d0264da9b452860716589ebafc0158941a034962149e554abb32e62"
Sep 11 17:33:07.827688 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:33:07.827637    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0293b2e5d0264da9b452860716589ebafc0158941a034962149e554abb32e62\": container with ID starting with b0293b2e5d0264da9b452860716589ebafc0158941a034962149e554abb32e62 not found: ID does not exist" containerID="b0293b2e5d0264da9b452860716589ebafc0158941a034962149e554abb32e62"
Sep 11 17:33:07.827688 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.827672    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0293b2e5d0264da9b452860716589ebafc0158941a034962149e554abb32e62"} err="failed to get container status \"b0293b2e5d0264da9b452860716589ebafc0158941a034962149e554abb32e62\": rpc error: code = NotFound desc = could not find container \"b0293b2e5d0264da9b452860716589ebafc0158941a034962149e554abb32e62\": container with ID starting with b0293b2e5d0264da9b452860716589ebafc0158941a034962149e554abb32e62 not found: ID does not exist"
Sep 11 17:33:07.829775 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.829751    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"]
Sep 11 17:33:07.833166 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:07.833148    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0ad997be-openshift-default-86dcbb565d-2778m"]
Sep 11 17:33:08.125830 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:08.125797    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ec7faa6-79e1-4e52-b823-55407f1a32d6" path="/var/lib/kubelet/pods/2ec7faa6-79e1-4e52-b823-55407f1a32d6/volumes"
Sep 11 17:33:10.225829 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:10.225795    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-357aed65-799bc6447d-4dmsq"]
Sep 11 17:33:10.226494 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:10.226169    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ec7faa6-79e1-4e52-b823-55407f1a32d6" containerName="istio-proxy"
Sep 11 17:33:10.226494 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:10.226180    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ec7faa6-79e1-4e52-b823-55407f1a32d6" containerName="istio-proxy"
Sep 11 17:33:10.226494 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:10.226190    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cd23851-76f5-4715-9c7a-f95dc7b075d7" containerName="payload-pre-processing"
Sep 11 17:33:10.226494 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:10.226195    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cd23851-76f5-4715-9c7a-f95dc7b075d7" containerName="payload-pre-processing"
Sep 11 17:33:10.226494 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:10.226212    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1000856f-12e0-4f2e-b3a7-1d39dc1d3b41" containerName="payload-processing"
Sep 11 17:33:10.226494 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:10.226217    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1000856f-12e0-4f2e-b3a7-1d39dc1d3b41" containerName="payload-processing"
Sep 11 17:33:10.226494 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:10.226276    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1000856f-12e0-4f2e-b3a7-1d39dc1d3b41" containerName="payload-processing"
Sep 11 17:33:10.226494 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:10.226284    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ec7faa6-79e1-4e52-b823-55407f1a32d6" containerName="istio-proxy"
Sep 11 17:33:10.226494 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:10.226292    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cd23851-76f5-4715-9c7a-f95dc7b075d7" containerName="payload-pre-processing"
Sep 11 17:33:10.226494 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:10.226299    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ec7faa6-79e1-4e52-b823-55407f1a32d6" containerName="istio-proxy"
Sep 11 17:33:10.229512 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:10.229487    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-357aed65-799bc6447d-4dmsq"
Sep 11 17:33:10.232098 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:10.232069    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-357aed65-dockercfg-6djvl\""
Sep 11 17:33:10.232194 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:10.232103    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-357aed65\""
Sep 11 17:33:10.241095 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:10.241060    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tggx\" (UniqueName: \"kubernetes.io/projected/02808ade-f136-458c-93bc-f47effcd82a4-kube-api-access-9tggx\") pod \"payload-pre-processing-e2e-derive-357aed65-799bc6447d-4dmsq\" (UID: \"02808ade-f136-458c-93bc-f47effcd82a4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-357aed65-799bc6447d-4dmsq"
Sep 11 17:33:10.241296 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:10.241269    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02808ade-f136-458c-93bc-f47effcd82a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-357aed65-799bc6447d-4dmsq\" (UID: \"02808ade-f136-458c-93bc-f47effcd82a4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-357aed65-799bc6447d-4dmsq"
Sep 11 17:33:10.242568 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:10.242545    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-357aed65-6684f46f4f-p7hkf"]
Sep 11 17:33:10.242978 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:10.242961    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ec7faa6-79e1-4e52-b823-55407f1a32d6" containerName="istio-proxy"
Sep 11 17:33:10.243073 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:10.242981    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ec7faa6-79e1-4e52-b823-55407f1a32d6" containerName="istio-proxy"
Sep 11 17:33:10.245813 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:10.245433    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-357aed65-6684f46f4f-p7hkf"
Sep 11 17:33:10.249204 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:10.249179    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-357aed65-799bc6447d-4dmsq"]
Sep 11 17:33:10.262238 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:10.262207    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-357aed65-6684f46f4f-p7hkf"]
Sep 11 17:33:10.342763 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:10.342722    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02808ade-f136-458c-93bc-f47effcd82a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-357aed65-799bc6447d-4dmsq\" (UID: \"02808ade-f136-458c-93bc-f47effcd82a4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-357aed65-799bc6447d-4dmsq"
Sep 11 17:33:10.342959 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:10.342793    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tggx\" (UniqueName: \"kubernetes.io/projected/02808ade-f136-458c-93bc-f47effcd82a4-kube-api-access-9tggx\") pod \"payload-pre-processing-e2e-derive-357aed65-799bc6447d-4dmsq\" (UID: \"02808ade-f136-458c-93bc-f47effcd82a4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-357aed65-799bc6447d-4dmsq"
Sep 11 17:33:10.342959 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:10.342917    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c6e1d5b-c9f7-4450-9b91-4bd48eaff9e0-plugins-config-volume\") pod \"payload-processing-e2e-derive-357aed65-6684f46f4f-p7hkf\" (UID: \"7c6e1d5b-c9f7-4450-9b91-4bd48eaff9e0\") " pod="openshift-ingress/payload-processing-e2e-derive-357aed65-6684f46f4f-p7hkf"
Sep 11 17:33:10.342959 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:10.342951    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6567h\" (UniqueName: \"kubernetes.io/projected/7c6e1d5b-c9f7-4450-9b91-4bd48eaff9e0-kube-api-access-6567h\") pod \"payload-processing-e2e-derive-357aed65-6684f46f4f-p7hkf\" (UID: \"7c6e1d5b-c9f7-4450-9b91-4bd48eaff9e0\") " pod="openshift-ingress/payload-processing-e2e-derive-357aed65-6684f46f4f-p7hkf"
Sep 11 17:33:10.343599 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:10.343573    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02808ade-f136-458c-93bc-f47effcd82a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-357aed65-799bc6447d-4dmsq\" (UID: \"02808ade-f136-458c-93bc-f47effcd82a4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-357aed65-799bc6447d-4dmsq"
Sep 11 17:33:10.351467 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:10.351443    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tggx\" (UniqueName: \"kubernetes.io/projected/02808ade-f136-458c-93bc-f47effcd82a4-kube-api-access-9tggx\") pod \"payload-pre-processing-e2e-derive-357aed65-799bc6447d-4dmsq\" (UID: \"02808ade-f136-458c-93bc-f47effcd82a4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-357aed65-799bc6447d-4dmsq"
Sep 11 17:33:10.444226 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:10.444188    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c6e1d5b-c9f7-4450-9b91-4bd48eaff9e0-plugins-config-volume\") pod \"payload-processing-e2e-derive-357aed65-6684f46f4f-p7hkf\" (UID: \"7c6e1d5b-c9f7-4450-9b91-4bd48eaff9e0\") " pod="openshift-ingress/payload-processing-e2e-derive-357aed65-6684f46f4f-p7hkf"
Sep 11 17:33:10.444226 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:10.444231    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6567h\" (UniqueName: \"kubernetes.io/projected/7c6e1d5b-c9f7-4450-9b91-4bd48eaff9e0-kube-api-access-6567h\") pod \"payload-processing-e2e-derive-357aed65-6684f46f4f-p7hkf\" (UID: \"7c6e1d5b-c9f7-4450-9b91-4bd48eaff9e0\") " pod="openshift-ingress/payload-processing-e2e-derive-357aed65-6684f46f4f-p7hkf"
Sep 11 17:33:10.444962 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:10.444943    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c6e1d5b-c9f7-4450-9b91-4bd48eaff9e0-plugins-config-volume\") pod \"payload-processing-e2e-derive-357aed65-6684f46f4f-p7hkf\" (UID: \"7c6e1d5b-c9f7-4450-9b91-4bd48eaff9e0\") " pod="openshift-ingress/payload-processing-e2e-derive-357aed65-6684f46f4f-p7hkf"
Sep 11 17:33:10.452644 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:10.452617    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6567h\" (UniqueName: \"kubernetes.io/projected/7c6e1d5b-c9f7-4450-9b91-4bd48eaff9e0-kube-api-access-6567h\") pod \"payload-processing-e2e-derive-357aed65-6684f46f4f-p7hkf\" (UID: \"7c6e1d5b-c9f7-4450-9b91-4bd48eaff9e0\") " pod="openshift-ingress/payload-processing-e2e-derive-357aed65-6684f46f4f-p7hkf"
Sep 11 17:33:10.541552 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:10.541467    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-357aed65-799bc6447d-4dmsq"
Sep 11 17:33:10.558615 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:10.558590    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-357aed65-6684f46f4f-p7hkf"
Sep 11 17:33:10.814599 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:10.814575    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-357aed65-799bc6447d-4dmsq"]
Sep 11 17:33:10.816761 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:33:10.816732    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02808ade_f136_458c_93bc_f47effcd82a4.slice/crio-446c731eab322bcb2218c3b2f15b0974cb53a6f317f8104bf0f4c7763669b842 WatchSource:0}: Error finding container 446c731eab322bcb2218c3b2f15b0974cb53a6f317f8104bf0f4c7763669b842: Status 404 returned error can't find the container with id 446c731eab322bcb2218c3b2f15b0974cb53a6f317f8104bf0f4c7763669b842
Sep 11 17:33:10.826283 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:10.826254    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-357aed65-799bc6447d-4dmsq" event={"ID":"02808ade-f136-458c-93bc-f47effcd82a4","Type":"ContainerStarted","Data":"446c731eab322bcb2218c3b2f15b0974cb53a6f317f8104bf0f4c7763669b842"}
Sep 11 17:33:10.838200 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:10.838174    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-357aed65-6684f46f4f-p7hkf"]
Sep 11 17:33:10.855193 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:33:10.855164    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c6e1d5b_c9f7_4450_9b91_4bd48eaff9e0.slice/crio-456a9de104cb11b06057e14802752ec1358aae9a41304a51475e0f1989f10bbf WatchSource:0}: Error finding container 456a9de104cb11b06057e14802752ec1358aae9a41304a51475e0f1989f10bbf: Status 404 returned error can't find the container with id 456a9de104cb11b06057e14802752ec1358aae9a41304a51475e0f1989f10bbf
Sep 11 17:33:11.835399 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:11.835362    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-357aed65-799bc6447d-4dmsq" event={"ID":"02808ade-f136-458c-93bc-f47effcd82a4","Type":"ContainerStarted","Data":"7da907412dc0273465a5156d268945c1c739038c3654df4597899fa6347ae216"}
Sep 11 17:33:11.835888 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:11.835426    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-357aed65-799bc6447d-4dmsq"
Sep 11 17:33:11.836891 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:11.836862    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-357aed65-6684f46f4f-p7hkf" event={"ID":"7c6e1d5b-c9f7-4450-9b91-4bd48eaff9e0","Type":"ContainerStarted","Data":"531befb01e703941bd390ecedd22a695215367088d8cb5603dea0ae6cecbf288"}
Sep 11 17:33:11.837027 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:11.836912    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-357aed65-6684f46f4f-p7hkf"
Sep 11 17:33:11.837027 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:11.836928    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-357aed65-6684f46f4f-p7hkf" event={"ID":"7c6e1d5b-c9f7-4450-9b91-4bd48eaff9e0","Type":"ContainerStarted","Data":"456a9de104cb11b06057e14802752ec1358aae9a41304a51475e0f1989f10bbf"}
Sep 11 17:33:11.850497 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:11.850444    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-357aed65-799bc6447d-4dmsq" podStartSLOduration=1.850427302 podStartE2EDuration="1.850427302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:33:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:33:11.849834668 +0000 UTC m=+872.243752124" watchObservedRunningTime="2026-09-11 17:33:11.850427302 +0000 UTC m=+872.244344759"
Sep 11 17:33:11.866583 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:11.866543    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-357aed65-6684f46f4f-p7hkf" podStartSLOduration=1.8665300500000002 podStartE2EDuration="1.86653005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:33:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:33:11.865608919 +0000 UTC m=+872.259526375" watchObservedRunningTime="2026-09-11 17:33:11.86653005 +0000 UTC m=+872.260447504"
Sep 11 17:33:22.843462 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:22.843430    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-357aed65-799bc6447d-4dmsq"
Sep 11 17:33:22.843985 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:22.843754    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-357aed65-6684f46f4f-p7hkf"
Sep 11 17:33:43.793406 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:43.793370    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-49e398/autores-model-49e398-kserve-c5c59fd95-d785v"]
Sep 11 17:33:43.796650 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:43.796633    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-49e398/autores-model-49e398-kserve-c5c59fd95-d785v"
Sep 11 17:33:43.799750 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:43.799730    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-49e398\"/\"kube-root-ca.crt\""
Sep 11 17:33:43.799750 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:43.799742    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-49e398\"/\"default-dockercfg-tzbxb\""
Sep 11 17:33:43.799961 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:43.799743    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-49e398\"/\"autores-model-49e398-kserve-self-signed-certs\""
Sep 11 17:33:43.799961 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:43.799801    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-49e398\"/\"openshift-service-ca.crt\""
Sep 11 17:33:43.806885 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:43.806864    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-49e398/autores-model-49e398-kserve-c5c59fd95-d785v"]
Sep 11 17:33:43.956405 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:43.956367    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4wgw\" (UniqueName: \"kubernetes.io/projected/dd301669-ba91-485d-9d0d-86c3291aff2a-kube-api-access-n4wgw\") pod \"autores-model-49e398-kserve-c5c59fd95-d785v\" (UID: \"dd301669-ba91-485d-9d0d-86c3291aff2a\") " pod="ai-tenant-e2e-autores-49e398/autores-model-49e398-kserve-c5c59fd95-d785v"
Sep 11 17:33:43.956575 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:43.956418    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd301669-ba91-485d-9d0d-86c3291aff2a-tmp-dir\") pod \"autores-model-49e398-kserve-c5c59fd95-d785v\" (UID: \"dd301669-ba91-485d-9d0d-86c3291aff2a\") " pod="ai-tenant-e2e-autores-49e398/autores-model-49e398-kserve-c5c59fd95-d785v"
Sep 11 17:33:43.956575 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:43.956445    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd301669-ba91-485d-9d0d-86c3291aff2a-home\") pod \"autores-model-49e398-kserve-c5c59fd95-d785v\" (UID: \"dd301669-ba91-485d-9d0d-86c3291aff2a\") " pod="ai-tenant-e2e-autores-49e398/autores-model-49e398-kserve-c5c59fd95-d785v"
Sep 11 17:33:43.956575 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:43.956506    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd301669-ba91-485d-9d0d-86c3291aff2a-dshm\") pod \"autores-model-49e398-kserve-c5c59fd95-d785v\" (UID: \"dd301669-ba91-485d-9d0d-86c3291aff2a\") " pod="ai-tenant-e2e-autores-49e398/autores-model-49e398-kserve-c5c59fd95-d785v"
Sep 11 17:33:43.956575 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:43.956540    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd301669-ba91-485d-9d0d-86c3291aff2a-tls-certs\") pod \"autores-model-49e398-kserve-c5c59fd95-d785v\" (UID: \"dd301669-ba91-485d-9d0d-86c3291aff2a\") " pod="ai-tenant-e2e-autores-49e398/autores-model-49e398-kserve-c5c59fd95-d785v"
Sep 11 17:33:43.956715 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:43.956631    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd301669-ba91-485d-9d0d-86c3291aff2a-model-cache\") pod \"autores-model-49e398-kserve-c5c59fd95-d785v\" (UID: \"dd301669-ba91-485d-9d0d-86c3291aff2a\") " pod="ai-tenant-e2e-autores-49e398/autores-model-49e398-kserve-c5c59fd95-d785v"
Sep 11 17:33:44.057576 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:44.057478    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd301669-ba91-485d-9d0d-86c3291aff2a-model-cache\") pod \"autores-model-49e398-kserve-c5c59fd95-d785v\" (UID: \"dd301669-ba91-485d-9d0d-86c3291aff2a\") " pod="ai-tenant-e2e-autores-49e398/autores-model-49e398-kserve-c5c59fd95-d785v"
Sep 11 17:33:44.057576 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:44.057532    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4wgw\" (UniqueName: \"kubernetes.io/projected/dd301669-ba91-485d-9d0d-86c3291aff2a-kube-api-access-n4wgw\") pod \"autores-model-49e398-kserve-c5c59fd95-d785v\" (UID: \"dd301669-ba91-485d-9d0d-86c3291aff2a\") " pod="ai-tenant-e2e-autores-49e398/autores-model-49e398-kserve-c5c59fd95-d785v"
Sep 11 17:33:44.057576 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:44.057572    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd301669-ba91-485d-9d0d-86c3291aff2a-tmp-dir\") pod \"autores-model-49e398-kserve-c5c59fd95-d785v\" (UID: \"dd301669-ba91-485d-9d0d-86c3291aff2a\") " pod="ai-tenant-e2e-autores-49e398/autores-model-49e398-kserve-c5c59fd95-d785v"
Sep 11 17:33:44.057846 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:44.057596    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd301669-ba91-485d-9d0d-86c3291aff2a-home\") pod \"autores-model-49e398-kserve-c5c59fd95-d785v\" (UID: \"dd301669-ba91-485d-9d0d-86c3291aff2a\") " pod="ai-tenant-e2e-autores-49e398/autores-model-49e398-kserve-c5c59fd95-d785v"
Sep 11 17:33:44.057846 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:44.057627    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd301669-ba91-485d-9d0d-86c3291aff2a-dshm\") pod \"autores-model-49e398-kserve-c5c59fd95-d785v\" (UID: \"dd301669-ba91-485d-9d0d-86c3291aff2a\") " pod="ai-tenant-e2e-autores-49e398/autores-model-49e398-kserve-c5c59fd95-d785v"
Sep 11 17:33:44.057846 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:44.057653    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd301669-ba91-485d-9d0d-86c3291aff2a-tls-certs\") pod \"autores-model-49e398-kserve-c5c59fd95-d785v\" (UID: \"dd301669-ba91-485d-9d0d-86c3291aff2a\") " pod="ai-tenant-e2e-autores-49e398/autores-model-49e398-kserve-c5c59fd95-d785v"
Sep 11 17:33:44.057976 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:44.057952    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd301669-ba91-485d-9d0d-86c3291aff2a-model-cache\") pod \"autores-model-49e398-kserve-c5c59fd95-d785v\" (UID: \"dd301669-ba91-485d-9d0d-86c3291aff2a\") " pod="ai-tenant-e2e-autores-49e398/autores-model-49e398-kserve-c5c59fd95-d785v"
Sep 11 17:33:44.058048 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:44.058018    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd301669-ba91-485d-9d0d-86c3291aff2a-tmp-dir\") pod \"autores-model-49e398-kserve-c5c59fd95-d785v\" (UID: \"dd301669-ba91-485d-9d0d-86c3291aff2a\") " pod="ai-tenant-e2e-autores-49e398/autores-model-49e398-kserve-c5c59fd95-d785v"
Sep 11 17:33:44.058089 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:44.058049    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd301669-ba91-485d-9d0d-86c3291aff2a-home\") pod \"autores-model-49e398-kserve-c5c59fd95-d785v\" (UID: \"dd301669-ba91-485d-9d0d-86c3291aff2a\") " pod="ai-tenant-e2e-autores-49e398/autores-model-49e398-kserve-c5c59fd95-d785v"
Sep 11 17:33:44.059936 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:44.059906    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd301669-ba91-485d-9d0d-86c3291aff2a-dshm\") pod \"autores-model-49e398-kserve-c5c59fd95-d785v\" (UID: \"dd301669-ba91-485d-9d0d-86c3291aff2a\") " pod="ai-tenant-e2e-autores-49e398/autores-model-49e398-kserve-c5c59fd95-d785v"
Sep 11 17:33:44.060184 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:44.060167    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd301669-ba91-485d-9d0d-86c3291aff2a-tls-certs\") pod \"autores-model-49e398-kserve-c5c59fd95-d785v\" (UID: \"dd301669-ba91-485d-9d0d-86c3291aff2a\") " pod="ai-tenant-e2e-autores-49e398/autores-model-49e398-kserve-c5c59fd95-d785v"
Sep 11 17:33:44.067103 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:44.067078    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4wgw\" (UniqueName: \"kubernetes.io/projected/dd301669-ba91-485d-9d0d-86c3291aff2a-kube-api-access-n4wgw\") pod \"autores-model-49e398-kserve-c5c59fd95-d785v\" (UID: \"dd301669-ba91-485d-9d0d-86c3291aff2a\") " pod="ai-tenant-e2e-autores-49e398/autores-model-49e398-kserve-c5c59fd95-d785v"
Sep 11 17:33:44.106043 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:44.106014    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-49e398/autores-model-49e398-kserve-c5c59fd95-d785v"
Sep 11 17:33:44.387779 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:44.387748    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-49e398/autores-model-49e398-kserve-c5c59fd95-d785v"]
Sep 11 17:33:44.391187 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:33:44.391159    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd301669_ba91_485d_9d0d_86c3291aff2a.slice/crio-1e121a9674107d177cb794049563253cd43709e3ddeb019537d1da92327091a9 WatchSource:0}: Error finding container 1e121a9674107d177cb794049563253cd43709e3ddeb019537d1da92327091a9: Status 404 returned error can't find the container with id 1e121a9674107d177cb794049563253cd43709e3ddeb019537d1da92327091a9
Sep 11 17:33:45.007583 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:45.007544    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-49e398/autores-model-49e398-kserve-c5c59fd95-d785v" event={"ID":"dd301669-ba91-485d-9d0d-86c3291aff2a","Type":"ContainerStarted","Data":"2ac39136f68a176ce2dfe1fff05600eebdf9f988c5d9f304b856930bc8b89e08"}
Sep 11 17:33:45.007583 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:45.007585    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-49e398/autores-model-49e398-kserve-c5c59fd95-d785v" event={"ID":"dd301669-ba91-485d-9d0d-86c3291aff2a","Type":"ContainerStarted","Data":"1e121a9674107d177cb794049563253cd43709e3ddeb019537d1da92327091a9"}
Sep 11 17:33:45.031237 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:45.031191    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-49e398/autores-model-49e398-kserve-c5c59fd95-d785v" podStartSLOduration=2.03117615 podStartE2EDuration="2.03117615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:33:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:33:45.029741026 +0000 UTC m=+905.423658481" watchObservedRunningTime="2026-09-11 17:33:45.03117615 +0000 UTC m=+905.425093641"
Sep 11 17:33:47.306634 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:47.306599    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-357aed65-6684f46f4f-p7hkf"]
Sep 11 17:33:47.307108 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:47.306858    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-357aed65-6684f46f4f-p7hkf" podUID="7c6e1d5b-c9f7-4450-9b91-4bd48eaff9e0" containerName="payload-processing" containerID="cri-o://531befb01e703941bd390ecedd22a695215367088d8cb5603dea0ae6cecbf288" gracePeriod=30
Sep 11 17:33:47.314157 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:47.314129    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-357aed65-799bc6447d-4dmsq"]
Sep 11 17:33:47.314703 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:47.314575    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-357aed65-799bc6447d-4dmsq" podUID="02808ade-f136-458c-93bc-f47effcd82a4" containerName="payload-pre-processing" containerID="cri-o://7da907412dc0273465a5156d268945c1c739038c3654df4597899fa6347ae216" gracePeriod=30
Sep 11 17:33:47.563690 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:47.563618    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-357aed65-6684f46f4f-p7hkf"
Sep 11 17:33:47.596136 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:47.596111    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-357aed65-799bc6447d-4dmsq"
Sep 11 17:33:47.693336 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:47.693305    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02808ade-f136-458c-93bc-f47effcd82a4-plugins-config-volume\") pod \"02808ade-f136-458c-93bc-f47effcd82a4\" (UID: \"02808ade-f136-458c-93bc-f47effcd82a4\") "
Sep 11 17:33:47.693520 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:47.693374    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6567h\" (UniqueName: \"kubernetes.io/projected/7c6e1d5b-c9f7-4450-9b91-4bd48eaff9e0-kube-api-access-6567h\") pod \"7c6e1d5b-c9f7-4450-9b91-4bd48eaff9e0\" (UID: \"7c6e1d5b-c9f7-4450-9b91-4bd48eaff9e0\") "
Sep 11 17:33:47.693520 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:47.693415    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tggx\" (UniqueName: \"kubernetes.io/projected/02808ade-f136-458c-93bc-f47effcd82a4-kube-api-access-9tggx\") pod \"02808ade-f136-458c-93bc-f47effcd82a4\" (UID: \"02808ade-f136-458c-93bc-f47effcd82a4\") "
Sep 11 17:33:47.693520 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:47.693459    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c6e1d5b-c9f7-4450-9b91-4bd48eaff9e0-plugins-config-volume\") pod \"7c6e1d5b-c9f7-4450-9b91-4bd48eaff9e0\" (UID: \"7c6e1d5b-c9f7-4450-9b91-4bd48eaff9e0\") "
Sep 11 17:33:47.693794 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:47.693743    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02808ade-f136-458c-93bc-f47effcd82a4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "02808ade-f136-458c-93bc-f47effcd82a4" (UID: "02808ade-f136-458c-93bc-f47effcd82a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:33:47.693936 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:47.693837    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02808ade-f136-458c-93bc-f47effcd82a4-plugins-config-volume\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:33:47.694184 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:47.694145    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c6e1d5b-c9f7-4450-9b91-4bd48eaff9e0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7c6e1d5b-c9f7-4450-9b91-4bd48eaff9e0" (UID: "7c6e1d5b-c9f7-4450-9b91-4bd48eaff9e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:33:47.698093 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:47.698069    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02808ade-f136-458c-93bc-f47effcd82a4-kube-api-access-9tggx" (OuterVolumeSpecName: "kube-api-access-9tggx") pod "02808ade-f136-458c-93bc-f47effcd82a4" (UID: "02808ade-f136-458c-93bc-f47effcd82a4"). InnerVolumeSpecName "kube-api-access-9tggx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:33:47.699448 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:47.699414    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c6e1d5b-c9f7-4450-9b91-4bd48eaff9e0-kube-api-access-6567h" (OuterVolumeSpecName: "kube-api-access-6567h") pod "7c6e1d5b-c9f7-4450-9b91-4bd48eaff9e0" (UID: "7c6e1d5b-c9f7-4450-9b91-4bd48eaff9e0"). InnerVolumeSpecName "kube-api-access-6567h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:33:47.795314 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:47.795282    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6567h\" (UniqueName: \"kubernetes.io/projected/7c6e1d5b-c9f7-4450-9b91-4bd48eaff9e0-kube-api-access-6567h\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:33:47.795314 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:47.795309    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tggx\" (UniqueName: \"kubernetes.io/projected/02808ade-f136-458c-93bc-f47effcd82a4-kube-api-access-9tggx\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:33:47.795314 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:47.795319    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c6e1d5b-c9f7-4450-9b91-4bd48eaff9e0-plugins-config-volume\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:33:48.023189 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:48.023156    2562 generic.go:356] "Generic (PLEG): container finished" podID="7c6e1d5b-c9f7-4450-9b91-4bd48eaff9e0" containerID="531befb01e703941bd390ecedd22a695215367088d8cb5603dea0ae6cecbf288" exitCode=0
Sep 11 17:33:48.023387 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:48.023193    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-357aed65-6684f46f4f-p7hkf"
Sep 11 17:33:48.023387 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:48.023243    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-357aed65-6684f46f4f-p7hkf" event={"ID":"7c6e1d5b-c9f7-4450-9b91-4bd48eaff9e0","Type":"ContainerDied","Data":"531befb01e703941bd390ecedd22a695215367088d8cb5603dea0ae6cecbf288"}
Sep 11 17:33:48.023387 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:48.023316    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-357aed65-6684f46f4f-p7hkf" event={"ID":"7c6e1d5b-c9f7-4450-9b91-4bd48eaff9e0","Type":"ContainerDied","Data":"456a9de104cb11b06057e14802752ec1358aae9a41304a51475e0f1989f10bbf"}
Sep 11 17:33:48.023387 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:48.023358    2562 scope.go:117] "RemoveContainer" containerID="531befb01e703941bd390ecedd22a695215367088d8cb5603dea0ae6cecbf288"
Sep 11 17:33:48.024619 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:48.024570    2562 generic.go:356] "Generic (PLEG): container finished" podID="02808ade-f136-458c-93bc-f47effcd82a4" containerID="7da907412dc0273465a5156d268945c1c739038c3654df4597899fa6347ae216" exitCode=0
Sep 11 17:33:48.024619 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:48.024609    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-357aed65-799bc6447d-4dmsq" event={"ID":"02808ade-f136-458c-93bc-f47effcd82a4","Type":"ContainerDied","Data":"7da907412dc0273465a5156d268945c1c739038c3654df4597899fa6347ae216"}
Sep 11 17:33:48.024733 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:48.024653    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-357aed65-799bc6447d-4dmsq" event={"ID":"02808ade-f136-458c-93bc-f47effcd82a4","Type":"ContainerDied","Data":"446c731eab322bcb2218c3b2f15b0974cb53a6f317f8104bf0f4c7763669b842"}
Sep 11 17:33:48.024733 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:48.024613    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-357aed65-799bc6447d-4dmsq"
Sep 11 17:33:48.032451 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:48.032393    2562 scope.go:117] "RemoveContainer" containerID="531befb01e703941bd390ecedd22a695215367088d8cb5603dea0ae6cecbf288"
Sep 11 17:33:48.032742 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:33:48.032721    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"531befb01e703941bd390ecedd22a695215367088d8cb5603dea0ae6cecbf288\": container with ID starting with 531befb01e703941bd390ecedd22a695215367088d8cb5603dea0ae6cecbf288 not found: ID does not exist" containerID="531befb01e703941bd390ecedd22a695215367088d8cb5603dea0ae6cecbf288"
Sep 11 17:33:48.032826 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:48.032747    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"531befb01e703941bd390ecedd22a695215367088d8cb5603dea0ae6cecbf288"} err="failed to get container status \"531befb01e703941bd390ecedd22a695215367088d8cb5603dea0ae6cecbf288\": rpc error: code = NotFound desc = could not find container \"531befb01e703941bd390ecedd22a695215367088d8cb5603dea0ae6cecbf288\": container with ID starting with 531befb01e703941bd390ecedd22a695215367088d8cb5603dea0ae6cecbf288 not found: ID does not exist"
Sep 11 17:33:48.032826 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:48.032769    2562 scope.go:117] "RemoveContainer" containerID="7da907412dc0273465a5156d268945c1c739038c3654df4597899fa6347ae216"
Sep 11 17:33:48.040483 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:48.040466    2562 scope.go:117] "RemoveContainer" containerID="7da907412dc0273465a5156d268945c1c739038c3654df4597899fa6347ae216"
Sep 11 17:33:48.040717 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:33:48.040695    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7da907412dc0273465a5156d268945c1c739038c3654df4597899fa6347ae216\": container with ID starting with 7da907412dc0273465a5156d268945c1c739038c3654df4597899fa6347ae216 not found: ID does not exist" containerID="7da907412dc0273465a5156d268945c1c739038c3654df4597899fa6347ae216"
Sep 11 17:33:48.040770 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:48.040724    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7da907412dc0273465a5156d268945c1c739038c3654df4597899fa6347ae216"} err="failed to get container status \"7da907412dc0273465a5156d268945c1c739038c3654df4597899fa6347ae216\": rpc error: code = NotFound desc = could not find container \"7da907412dc0273465a5156d268945c1c739038c3654df4597899fa6347ae216\": container with ID starting with 7da907412dc0273465a5156d268945c1c739038c3654df4597899fa6347ae216 not found: ID does not exist"
Sep 11 17:33:48.059160 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:48.059137    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-357aed65-6684f46f4f-p7hkf"]
Sep 11 17:33:48.076984 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:48.076960    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-357aed65-6684f46f4f-p7hkf"]
Sep 11 17:33:48.105820 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:48.105800    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-357aed65-799bc6447d-4dmsq"]
Sep 11 17:33:48.114361 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:48.114338    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-357aed65-799bc6447d-4dmsq"]
Sep 11 17:33:48.125610 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:48.125589    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02808ade-f136-458c-93bc-f47effcd82a4" path="/var/lib/kubelet/pods/02808ade-f136-458c-93bc-f47effcd82a4/volumes"
Sep 11 17:33:48.125965 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:48.125951    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c6e1d5b-c9f7-4450-9b91-4bd48eaff9e0" path="/var/lib/kubelet/pods/7c6e1d5b-c9f7-4450-9b91-4bd48eaff9e0/volumes"
Sep 11 17:33:54.106813 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:54.106774    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-49e398/autores-model-49e398-kserve-c5c59fd95-d785v"
Sep 11 17:33:54.107276 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:54.106866    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-49e398/autores-model-49e398-kserve-c5c59fd95-d785v"
Sep 11 17:33:54.126265 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:54.126238    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-49e398/autores-model-49e398-kserve-c5c59fd95-d785v"
Sep 11 17:33:55.071432 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:33:55.071401    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-49e398/autores-model-49e398-kserve-c5c59fd95-d785v"
Sep 11 17:34:26.944543 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:26.944501    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-49e398/autores-model-49e398-kserve-c5c59fd95-d785v"]
Sep 11 17:34:30.012832 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:30.012796    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-49e398-567f8655bf-qf2pj"]
Sep 11 17:34:30.013282 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:30.013068    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-49e398-567f8655bf-qf2pj" podUID="6c1c624d-7469-4934-9214-fc400af5de05" containerName="payload-processing" containerID="cri-o://8a7aa1820b48d5dda5e03f9aeaf3e37246313e7bb1348a2456fd185fa683af45" gracePeriod=30
Sep 11 17:34:30.236028 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:30.235972    2562 generic.go:356] "Generic (PLEG): container finished" podID="6c1c624d-7469-4934-9214-fc400af5de05" containerID="8a7aa1820b48d5dda5e03f9aeaf3e37246313e7bb1348a2456fd185fa683af45" exitCode=0
Sep 11 17:34:30.236221 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:30.236034    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-49e398-567f8655bf-qf2pj" event={"ID":"6c1c624d-7469-4934-9214-fc400af5de05","Type":"ContainerDied","Data":"8a7aa1820b48d5dda5e03f9aeaf3e37246313e7bb1348a2456fd185fa683af45"}
Sep 11 17:34:30.269187 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:30.269126    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-49e398-567f8655bf-qf2pj"
Sep 11 17:34:30.389412 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:30.389383    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1c624d-7469-4934-9214-fc400af5de05-plugins-config-volume\") pod \"6c1c624d-7469-4934-9214-fc400af5de05\" (UID: \"6c1c624d-7469-4934-9214-fc400af5de05\") "
Sep 11 17:34:30.389612 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:30.389443    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4plxs\" (UniqueName: \"kubernetes.io/projected/6c1c624d-7469-4934-9214-fc400af5de05-kube-api-access-4plxs\") pod \"6c1c624d-7469-4934-9214-fc400af5de05\" (UID: \"6c1c624d-7469-4934-9214-fc400af5de05\") "
Sep 11 17:34:30.389850 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:30.389816    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c1c624d-7469-4934-9214-fc400af5de05-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6c1c624d-7469-4934-9214-fc400af5de05" (UID: "6c1c624d-7469-4934-9214-fc400af5de05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:34:30.391623 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:30.391593    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c1c624d-7469-4934-9214-fc400af5de05-kube-api-access-4plxs" (OuterVolumeSpecName: "kube-api-access-4plxs") pod "6c1c624d-7469-4934-9214-fc400af5de05" (UID: "6c1c624d-7469-4934-9214-fc400af5de05"). InnerVolumeSpecName "kube-api-access-4plxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:34:30.490809 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:30.490778    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1c624d-7469-4934-9214-fc400af5de05-plugins-config-volume\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:34:30.490809 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:30.490803    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4plxs\" (UniqueName: \"kubernetes.io/projected/6c1c624d-7469-4934-9214-fc400af5de05-kube-api-access-4plxs\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:34:31.242201 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:31.242165    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-49e398-567f8655bf-qf2pj" event={"ID":"6c1c624d-7469-4934-9214-fc400af5de05","Type":"ContainerDied","Data":"02aa6436c329e944b9c05c9ecc868e061feafdd9d79ae98df23747cbd622bf3c"}
Sep 11 17:34:31.242201 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:31.242186    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-49e398-567f8655bf-qf2pj"
Sep 11 17:34:31.242742 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:31.242229    2562 scope.go:117] "RemoveContainer" containerID="8a7aa1820b48d5dda5e03f9aeaf3e37246313e7bb1348a2456fd185fa683af45"
Sep 11 17:34:31.267055 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:31.267024    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-49e398-567f8655bf-qf2pj"]
Sep 11 17:34:31.270780 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:31.270757    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-49e398-567f8655bf-qf2pj"]
Sep 11 17:34:32.125924 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:32.125884    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c1c624d-7469-4934-9214-fc400af5de05" path="/var/lib/kubelet/pods/6c1c624d-7469-4934-9214-fc400af5de05/volumes"
Sep 11 17:34:41.964819 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:41.964752    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-49e398/autores-model-49e398-kserve-c5c59fd95-d785v" podUID="dd301669-ba91-485d-9d0d-86c3291aff2a" containerName="main" containerID="cri-o://2ac39136f68a176ce2dfe1fff05600eebdf9f988c5d9f304b856930bc8b89e08" gracePeriod=45
Sep 11 17:34:42.223668 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:42.223609    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-49e398/autores-model-49e398-kserve-c5c59fd95-d785v"
Sep 11 17:34:42.291701 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:42.291658    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd301669-ba91-485d-9d0d-86c3291aff2a-home\") pod \"dd301669-ba91-485d-9d0d-86c3291aff2a\" (UID: \"dd301669-ba91-485d-9d0d-86c3291aff2a\") "
Sep 11 17:34:42.291894 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:42.291730    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4wgw\" (UniqueName: \"kubernetes.io/projected/dd301669-ba91-485d-9d0d-86c3291aff2a-kube-api-access-n4wgw\") pod \"dd301669-ba91-485d-9d0d-86c3291aff2a\" (UID: \"dd301669-ba91-485d-9d0d-86c3291aff2a\") "
Sep 11 17:34:42.291894 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:42.291759    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd301669-ba91-485d-9d0d-86c3291aff2a-tmp-dir\") pod \"dd301669-ba91-485d-9d0d-86c3291aff2a\" (UID: \"dd301669-ba91-485d-9d0d-86c3291aff2a\") "
Sep 11 17:34:42.291894 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:42.291843    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd301669-ba91-485d-9d0d-86c3291aff2a-dshm\") pod \"dd301669-ba91-485d-9d0d-86c3291aff2a\" (UID: \"dd301669-ba91-485d-9d0d-86c3291aff2a\") "
Sep 11 17:34:42.291894 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:42.291865    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd301669-ba91-485d-9d0d-86c3291aff2a-tls-certs\") pod \"dd301669-ba91-485d-9d0d-86c3291aff2a\" (UID: \"dd301669-ba91-485d-9d0d-86c3291aff2a\") "
Sep 11 17:34:42.291894 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:42.291893    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd301669-ba91-485d-9d0d-86c3291aff2a-model-cache\") pod \"dd301669-ba91-485d-9d0d-86c3291aff2a\" (UID: \"dd301669-ba91-485d-9d0d-86c3291aff2a\") "
Sep 11 17:34:42.292222 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:42.291914    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd301669-ba91-485d-9d0d-86c3291aff2a-home" (OuterVolumeSpecName: "home") pod "dd301669-ba91-485d-9d0d-86c3291aff2a" (UID: "dd301669-ba91-485d-9d0d-86c3291aff2a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:34:42.292222 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:42.292099    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd301669-ba91-485d-9d0d-86c3291aff2a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dd301669-ba91-485d-9d0d-86c3291aff2a" (UID: "dd301669-ba91-485d-9d0d-86c3291aff2a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:34:42.292315 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:42.292243    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd301669-ba91-485d-9d0d-86c3291aff2a-model-cache" (OuterVolumeSpecName: "model-cache") pod "dd301669-ba91-485d-9d0d-86c3291aff2a" (UID: "dd301669-ba91-485d-9d0d-86c3291aff2a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:34:42.292365 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:42.292318    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd301669-ba91-485d-9d0d-86c3291aff2a-model-cache\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:34:42.292365 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:42.292337    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd301669-ba91-485d-9d0d-86c3291aff2a-home\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:34:42.292494 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:42.292372    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd301669-ba91-485d-9d0d-86c3291aff2a-tmp-dir\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:34:42.294155 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:42.294127    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd301669-ba91-485d-9d0d-86c3291aff2a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dd301669-ba91-485d-9d0d-86c3291aff2a" (UID: "dd301669-ba91-485d-9d0d-86c3291aff2a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:34:42.294264 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:42.294135    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd301669-ba91-485d-9d0d-86c3291aff2a-dshm" (OuterVolumeSpecName: "dshm") pod "dd301669-ba91-485d-9d0d-86c3291aff2a" (UID: "dd301669-ba91-485d-9d0d-86c3291aff2a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:34:42.294264 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:42.294181    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd301669-ba91-485d-9d0d-86c3291aff2a-kube-api-access-n4wgw" (OuterVolumeSpecName: "kube-api-access-n4wgw") pod "dd301669-ba91-485d-9d0d-86c3291aff2a" (UID: "dd301669-ba91-485d-9d0d-86c3291aff2a"). InnerVolumeSpecName "kube-api-access-n4wgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:34:42.301298 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:42.301271    2562 generic.go:356] "Generic (PLEG): container finished" podID="dd301669-ba91-485d-9d0d-86c3291aff2a" containerID="2ac39136f68a176ce2dfe1fff05600eebdf9f988c5d9f304b856930bc8b89e08" exitCode=0
Sep 11 17:34:42.301416 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:42.301310    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-49e398/autores-model-49e398-kserve-c5c59fd95-d785v"
Sep 11 17:34:42.301416 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:42.301351    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-49e398/autores-model-49e398-kserve-c5c59fd95-d785v" event={"ID":"dd301669-ba91-485d-9d0d-86c3291aff2a","Type":"ContainerDied","Data":"2ac39136f68a176ce2dfe1fff05600eebdf9f988c5d9f304b856930bc8b89e08"}
Sep 11 17:34:42.301416 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:42.301397    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-49e398/autores-model-49e398-kserve-c5c59fd95-d785v" event={"ID":"dd301669-ba91-485d-9d0d-86c3291aff2a","Type":"ContainerDied","Data":"1e121a9674107d177cb794049563253cd43709e3ddeb019537d1da92327091a9"}
Sep 11 17:34:42.301416 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:42.301414    2562 scope.go:117] "RemoveContainer" containerID="2ac39136f68a176ce2dfe1fff05600eebdf9f988c5d9f304b856930bc8b89e08"
Sep 11 17:34:42.316055 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:42.316031    2562 scope.go:117] "RemoveContainer" containerID="2ac39136f68a176ce2dfe1fff05600eebdf9f988c5d9f304b856930bc8b89e08"
Sep 11 17:34:42.316366 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:34:42.316339    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ac39136f68a176ce2dfe1fff05600eebdf9f988c5d9f304b856930bc8b89e08\": container with ID starting with 2ac39136f68a176ce2dfe1fff05600eebdf9f988c5d9f304b856930bc8b89e08 not found: ID does not exist" containerID="2ac39136f68a176ce2dfe1fff05600eebdf9f988c5d9f304b856930bc8b89e08"
Sep 11 17:34:42.316435 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:42.316378    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ac39136f68a176ce2dfe1fff05600eebdf9f988c5d9f304b856930bc8b89e08"} err="failed to get container status \"2ac39136f68a176ce2dfe1fff05600eebdf9f988c5d9f304b856930bc8b89e08\": rpc error: code = NotFound desc = could not find container \"2ac39136f68a176ce2dfe1fff05600eebdf9f988c5d9f304b856930bc8b89e08\": container with ID starting with 2ac39136f68a176ce2dfe1fff05600eebdf9f988c5d9f304b856930bc8b89e08 not found: ID does not exist"
Sep 11 17:34:42.325871 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:42.325845    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-49e398/autores-model-49e398-kserve-c5c59fd95-d785v"]
Sep 11 17:34:42.327559 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:42.327539    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-49e398/autores-model-49e398-kserve-c5c59fd95-d785v"]
Sep 11 17:34:42.393350 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:42.393319    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4wgw\" (UniqueName: \"kubernetes.io/projected/dd301669-ba91-485d-9d0d-86c3291aff2a-kube-api-access-n4wgw\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:34:42.393350 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:42.393344    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd301669-ba91-485d-9d0d-86c3291aff2a-dshm\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:34:42.393350 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:42.393355    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd301669-ba91-485d-9d0d-86c3291aff2a-tls-certs\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:34:44.126132 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:34:44.126097    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd301669-ba91-485d-9d0d-86c3291aff2a" path="/var/lib/kubelet/pods/dd301669-ba91-485d-9d0d-86c3291aff2a/volumes"
Sep 11 17:35:20.664073 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:20.664023    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ec3fed-7865cdbb49-4dpfb"]
Sep 11 17:35:20.664506 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:20.664418    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd301669-ba91-485d-9d0d-86c3291aff2a" containerName="main"
Sep 11 17:35:20.664506 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:20.664430    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd301669-ba91-485d-9d0d-86c3291aff2a" containerName="main"
Sep 11 17:35:20.664506 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:20.664449    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c6e1d5b-c9f7-4450-9b91-4bd48eaff9e0" containerName="payload-processing"
Sep 11 17:35:20.664506 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:20.664455    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c6e1d5b-c9f7-4450-9b91-4bd48eaff9e0" containerName="payload-processing"
Sep 11 17:35:20.664506 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:20.664468    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c1c624d-7469-4934-9214-fc400af5de05" containerName="payload-processing"
Sep 11 17:35:20.664506 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:20.664474    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c1c624d-7469-4934-9214-fc400af5de05" containerName="payload-processing"
Sep 11 17:35:20.664506 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:20.664482    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02808ade-f136-458c-93bc-f47effcd82a4" containerName="payload-pre-processing"
Sep 11 17:35:20.664506 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:20.664487    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="02808ade-f136-458c-93bc-f47effcd82a4" containerName="payload-pre-processing"
Sep 11 17:35:20.664755 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:20.664543    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd301669-ba91-485d-9d0d-86c3291aff2a" containerName="main"
Sep 11 17:35:20.664755 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:20.664553    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="02808ade-f136-458c-93bc-f47effcd82a4" containerName="payload-pre-processing"
Sep 11 17:35:20.664755 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:20.664559    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c6e1d5b-c9f7-4450-9b91-4bd48eaff9e0" containerName="payload-processing"
Sep 11 17:35:20.664755 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:20.664566    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c1c624d-7469-4934-9214-fc400af5de05" containerName="payload-processing"
Sep 11 17:35:20.667488 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:20.667465    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-ec3fed-7865cdbb49-4dpfb"
Sep 11 17:35:20.669760 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:20.669739    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-ec3fed\""
Sep 11 17:35:20.683065 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:20.683043    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ec3fed-7865cdbb49-4dpfb"]
Sep 11 17:35:20.727734 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:20.727705    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ec3fed-567c4fd854-rt5x7"]
Sep 11 17:35:20.730760 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:20.730743    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ec3fed-567c4fd854-rt5x7"
Sep 11 17:35:20.733179 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:20.733161    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-ec3fed-dockercfg-xtn8x\""
Sep 11 17:35:20.741566 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:20.741542    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ec3fed-567c4fd854-rt5x7"]
Sep 11 17:35:20.748749 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:20.748725    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ffb5bd3-cb59-415a-b061-6084950220de-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-ec3fed-567c4fd854-rt5x7\" (UID: \"9ffb5bd3-cb59-415a-b061-6084950220de\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ec3fed-567c4fd854-rt5x7"
Sep 11 17:35:20.748867 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:20.748785    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31921d54-4a87-46f4-aed5-4954c6946bd3-plugins-config-volume\") pod \"payload-processing-e2e-explicit-ec3fed-7865cdbb49-4dpfb\" (UID: \"31921d54-4a87-46f4-aed5-4954c6946bd3\") " pod="openshift-ingress/payload-processing-e2e-explicit-ec3fed-7865cdbb49-4dpfb"
Sep 11 17:35:20.748867 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:20.748813    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h46wb\" (UniqueName: \"kubernetes.io/projected/31921d54-4a87-46f4-aed5-4954c6946bd3-kube-api-access-h46wb\") pod \"payload-processing-e2e-explicit-ec3fed-7865cdbb49-4dpfb\" (UID: \"31921d54-4a87-46f4-aed5-4954c6946bd3\") " pod="openshift-ingress/payload-processing-e2e-explicit-ec3fed-7865cdbb49-4dpfb"
Sep 11 17:35:20.748953 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:20.748898    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfbn4\" (UniqueName: \"kubernetes.io/projected/9ffb5bd3-cb59-415a-b061-6084950220de-kube-api-access-zfbn4\") pod \"payload-pre-processing-e2e-explicit-ec3fed-567c4fd854-rt5x7\" (UID: \"9ffb5bd3-cb59-415a-b061-6084950220de\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ec3fed-567c4fd854-rt5x7"
Sep 11 17:35:20.849862 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:20.849825    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ffb5bd3-cb59-415a-b061-6084950220de-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-ec3fed-567c4fd854-rt5x7\" (UID: \"9ffb5bd3-cb59-415a-b061-6084950220de\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ec3fed-567c4fd854-rt5x7"
Sep 11 17:35:20.849862 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:20.849883    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31921d54-4a87-46f4-aed5-4954c6946bd3-plugins-config-volume\") pod \"payload-processing-e2e-explicit-ec3fed-7865cdbb49-4dpfb\" (UID: \"31921d54-4a87-46f4-aed5-4954c6946bd3\") " pod="openshift-ingress/payload-processing-e2e-explicit-ec3fed-7865cdbb49-4dpfb"
Sep 11 17:35:20.850146 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:20.849900    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h46wb\" (UniqueName: \"kubernetes.io/projected/31921d54-4a87-46f4-aed5-4954c6946bd3-kube-api-access-h46wb\") pod \"payload-processing-e2e-explicit-ec3fed-7865cdbb49-4dpfb\" (UID: \"31921d54-4a87-46f4-aed5-4954c6946bd3\") " pod="openshift-ingress/payload-processing-e2e-explicit-ec3fed-7865cdbb49-4dpfb"
Sep 11 17:35:20.850146 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:20.849924    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfbn4\" (UniqueName: \"kubernetes.io/projected/9ffb5bd3-cb59-415a-b061-6084950220de-kube-api-access-zfbn4\") pod \"payload-pre-processing-e2e-explicit-ec3fed-567c4fd854-rt5x7\" (UID: \"9ffb5bd3-cb59-415a-b061-6084950220de\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ec3fed-567c4fd854-rt5x7"
Sep 11 17:35:20.850648 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:20.850625    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ffb5bd3-cb59-415a-b061-6084950220de-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-ec3fed-567c4fd854-rt5x7\" (UID: \"9ffb5bd3-cb59-415a-b061-6084950220de\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ec3fed-567c4fd854-rt5x7"
Sep 11 17:35:20.850769 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:20.850747    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31921d54-4a87-46f4-aed5-4954c6946bd3-plugins-config-volume\") pod \"payload-processing-e2e-explicit-ec3fed-7865cdbb49-4dpfb\" (UID: \"31921d54-4a87-46f4-aed5-4954c6946bd3\") " pod="openshift-ingress/payload-processing-e2e-explicit-ec3fed-7865cdbb49-4dpfb"
Sep 11 17:35:20.859409 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:20.859375    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h46wb\" (UniqueName: \"kubernetes.io/projected/31921d54-4a87-46f4-aed5-4954c6946bd3-kube-api-access-h46wb\") pod \"payload-processing-e2e-explicit-ec3fed-7865cdbb49-4dpfb\" (UID: \"31921d54-4a87-46f4-aed5-4954c6946bd3\") " pod="openshift-ingress/payload-processing-e2e-explicit-ec3fed-7865cdbb49-4dpfb"
Sep 11 17:35:20.859559 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:20.859455    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfbn4\" (UniqueName: \"kubernetes.io/projected/9ffb5bd3-cb59-415a-b061-6084950220de-kube-api-access-zfbn4\") pod \"payload-pre-processing-e2e-explicit-ec3fed-567c4fd854-rt5x7\" (UID: \"9ffb5bd3-cb59-415a-b061-6084950220de\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ec3fed-567c4fd854-rt5x7"
Sep 11 17:35:20.977491 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:20.977402    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-ec3fed-7865cdbb49-4dpfb"
Sep 11 17:35:21.041691 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:21.041369    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ec3fed-567c4fd854-rt5x7"
Sep 11 17:35:21.251813 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:21.251789    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ec3fed-7865cdbb49-4dpfb"]
Sep 11 17:35:21.254192 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:35:21.254159    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31921d54_4a87_46f4_aed5_4954c6946bd3.slice/crio-ffc7aa8b105067a20c2cd46eccc152bc521e5102ffd4ae354f9c2be3a3ec4d93 WatchSource:0}: Error finding container ffc7aa8b105067a20c2cd46eccc152bc521e5102ffd4ae354f9c2be3a3ec4d93: Status 404 returned error can't find the container with id ffc7aa8b105067a20c2cd46eccc152bc521e5102ffd4ae354f9c2be3a3ec4d93
Sep 11 17:35:21.314379 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:21.314347    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ec3fed-567c4fd854-rt5x7"]
Sep 11 17:35:21.316036 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:35:21.316012    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ffb5bd3_cb59_415a_b061_6084950220de.slice/crio-d96f49271c2dc5e55b28856f7177b1a5652305d3b869ed42b6e66a0fd9a3dd3a WatchSource:0}: Error finding container d96f49271c2dc5e55b28856f7177b1a5652305d3b869ed42b6e66a0fd9a3dd3a: Status 404 returned error can't find the container with id d96f49271c2dc5e55b28856f7177b1a5652305d3b869ed42b6e66a0fd9a3dd3a
Sep 11 17:35:21.488220 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:21.487942    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ec3fed-567c4fd854-rt5x7" event={"ID":"9ffb5bd3-cb59-415a-b061-6084950220de","Type":"ContainerStarted","Data":"510aa455c9bd7d8c1199b82a8d8b55eeff2f396060775118f139f02e283e8839"}
Sep 11 17:35:21.488220 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:21.488133    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ec3fed-567c4fd854-rt5x7"
Sep 11 17:35:21.488220 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:21.488174    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ec3fed-567c4fd854-rt5x7" event={"ID":"9ffb5bd3-cb59-415a-b061-6084950220de","Type":"ContainerStarted","Data":"d96f49271c2dc5e55b28856f7177b1a5652305d3b869ed42b6e66a0fd9a3dd3a"}
Sep 11 17:35:21.489878 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:21.489853    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-ec3fed-7865cdbb49-4dpfb" event={"ID":"31921d54-4a87-46f4-aed5-4954c6946bd3","Type":"ContainerStarted","Data":"bf17fbcf2911bc01d8dfe9ff3696d3b5d24dc3c5c2492ec4191a5f2b13c7fea4"}
Sep 11 17:35:21.490023 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:21.489897    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-ec3fed-7865cdbb49-4dpfb" event={"ID":"31921d54-4a87-46f4-aed5-4954c6946bd3","Type":"ContainerStarted","Data":"ffc7aa8b105067a20c2cd46eccc152bc521e5102ffd4ae354f9c2be3a3ec4d93"}
Sep 11 17:35:21.490078 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:21.490043    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-ec3fed-7865cdbb49-4dpfb"
Sep 11 17:35:21.507903 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:21.507806    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ec3fed-567c4fd854-rt5x7" podStartSLOduration=1.507791898 podStartE2EDuration="1.507791898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:35:21.506318508 +0000 UTC m=+1001.900235967" watchObservedRunningTime="2026-09-11 17:35:21.507791898 +0000 UTC m=+1001.901709352"
Sep 11 17:35:21.521984 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:21.521931    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-ec3fed-7865cdbb49-4dpfb" podStartSLOduration=1.521912719 podStartE2EDuration="1.521912719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:35:21.521515294 +0000 UTC m=+1001.915432748" watchObservedRunningTime="2026-09-11 17:35:21.521912719 +0000 UTC m=+1001.915830177"
Sep 11 17:35:23.266136 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:23.266099    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"]
Sep 11 17:35:23.270181 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:23.270154    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"
Sep 11 17:35:23.273350 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:23.273323    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-65fdd4fd-openshift-default-dockercfg-5b7nj\""
Sep 11 17:35:23.286773 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:23.286736    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"]
Sep 11 17:35:23.373928 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:23.373881    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8klb\" (UniqueName: \"kubernetes.io/projected/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-kube-api-access-z8klb\") pod \"e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66\" (UID: \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\") " pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"
Sep 11 17:35:23.374130 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:23.373964    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-istio-token\") pod \"e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66\" (UID: \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\") " pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"
Sep 11 17:35:23.374130 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:23.374061    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-workload-socket\") pod \"e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66\" (UID: \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\") " pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"
Sep 11 17:35:23.374130 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:23.374090    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-istio-data\") pod \"e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66\" (UID: \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\") " pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"
Sep 11 17:35:23.374130 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:23.374125    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-credential-socket\") pod \"e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66\" (UID: \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\") " pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"
Sep 11 17:35:23.374378 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:23.374154    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-istiod-ca-cert\") pod \"e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66\" (UID: \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\") " pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"
Sep 11 17:35:23.374378 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:23.374179    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-workload-certs\") pod \"e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66\" (UID: \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\") " pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"
Sep 11 17:35:23.374378 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:23.374220    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-istio-envoy\") pod \"e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66\" (UID: \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\") " pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"
Sep 11 17:35:23.374378 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:23.374243    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-istio-podinfo\") pod \"e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66\" (UID: \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\") " pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"
Sep 11 17:35:23.475651 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:23.475619    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-workload-socket\") pod \"e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66\" (UID: \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\") " pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"
Sep 11 17:35:23.475850 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:23.475657    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-istio-data\") pod \"e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66\" (UID: \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\") " pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"
Sep 11 17:35:23.475850 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:23.475699    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-credential-socket\") pod \"e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66\" (UID: \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\") " pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"
Sep 11 17:35:23.475850 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:23.475721    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-istiod-ca-cert\") pod \"e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66\" (UID: \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\") " pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"
Sep 11 17:35:23.475850 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:23.475749    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-workload-certs\") pod \"e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66\" (UID: \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\") " pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"
Sep 11 17:35:23.475850 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:23.475804    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-istio-envoy\") pod \"e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66\" (UID: \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\") " pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"
Sep 11 17:35:23.475850 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:23.475826    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-istio-podinfo\") pod \"e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66\" (UID: \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\") " pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"
Sep 11 17:35:23.476185 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:23.475904    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8klb\" (UniqueName: \"kubernetes.io/projected/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-kube-api-access-z8klb\") pod \"e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66\" (UID: \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\") " pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"
Sep 11 17:35:23.476185 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:23.475959    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-istio-token\") pod \"e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66\" (UID: \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\") " pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"
Sep 11 17:35:23.476185 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:23.476103    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-workload-socket\") pod \"e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66\" (UID: \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\") " pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"
Sep 11 17:35:23.476185 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:23.476131    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-workload-certs\") pod \"e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66\" (UID: \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\") " pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"
Sep 11 17:35:23.476429 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:23.476380    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-istio-data\") pod \"e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66\" (UID: \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\") " pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"
Sep 11 17:35:23.476561 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:23.476453    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-credential-socket\") pod \"e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66\" (UID: \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\") " pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"
Sep 11 17:35:23.476561 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:23.476513    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-istiod-ca-cert\") pod \"e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66\" (UID: \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\") " pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"
Sep 11 17:35:23.478394 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:23.478370    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-istio-envoy\") pod \"e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66\" (UID: \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\") " pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"
Sep 11 17:35:23.478494 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:23.478467    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-istio-podinfo\") pod \"e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66\" (UID: \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\") " pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"
Sep 11 17:35:23.483290 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:23.483259    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-istio-token\") pod \"e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66\" (UID: \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\") " pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"
Sep 11 17:35:23.483619 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:23.483599    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8klb\" (UniqueName: \"kubernetes.io/projected/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-kube-api-access-z8klb\") pod \"e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66\" (UID: \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\") " pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"
Sep 11 17:35:23.584903 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:23.584810    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"
Sep 11 17:35:23.945522 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:23.945488    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"]
Sep 11 17:35:23.948976 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:35:23.948938    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea06effd_49f5_4ae9_93c8_eb1f56b13a0e.slice/crio-c94446b9ab0d6dcb6531b551f4148eb240ce61ad2e8b04c345a0c77d257667ae WatchSource:0}: Error finding container c94446b9ab0d6dcb6531b551f4148eb240ce61ad2e8b04c345a0c77d257667ae: Status 404 returned error can't find the container with id c94446b9ab0d6dcb6531b551f4148eb240ce61ad2e8b04c345a0c77d257667ae
Sep 11 17:35:23.952062 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:23.952024    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:35:23.952166 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:23.952110    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:35:23.952166 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:23.952152    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:35:24.506103 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:24.506067    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66" event={"ID":"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e","Type":"ContainerStarted","Data":"c8559e81378e89db4184ab16d081aae34ac4dbe92117a37cb265bf23352937be"}
Sep 11 17:35:24.506478 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:24.506131    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66" event={"ID":"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e","Type":"ContainerStarted","Data":"c94446b9ab0d6dcb6531b551f4148eb240ce61ad2e8b04c345a0c77d257667ae"}
Sep 11 17:35:24.530796 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:24.530746    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66" podStartSLOduration=1.530729107 podStartE2EDuration="1.530729107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:35:24.529269679 +0000 UTC m=+1004.923187137" watchObservedRunningTime="2026-09-11 17:35:24.530729107 +0000 UTC m=+1004.924646562"
Sep 11 17:35:24.585107 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:24.585079    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"
Sep 11 17:35:25.590618 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:25.590588    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"
Sep 11 17:35:26.515770 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:26.515730    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"
Sep 11 17:35:26.516736 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:26.516715    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"
Sep 11 17:35:32.497536 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:32.497504    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-ec3fed-7865cdbb49-4dpfb"
Sep 11 17:35:32.498027 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:32.497700    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ec3fed-567c4fd854-rt5x7"
Sep 11 17:35:34.838317 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:34.838280    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-65fdd4fd-77857ffd69-7jx4z"]
Sep 11 17:35:34.842110 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:34.842088    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-65fdd4fd-77857ffd69-7jx4z"
Sep 11 17:35:34.847157 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:34.847131    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-65fdd4fd\""
Sep 11 17:35:34.847157 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:34.847154    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-65fdd4fd-dockercfg-kmlrm\""
Sep 11 17:35:34.869514 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:34.869479    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-65fdd4fd-77857ffd69-7jx4z"]
Sep 11 17:35:34.890701 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:34.890663    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtgs6\" (UniqueName: \"kubernetes.io/projected/73d34346-3963-480f-9081-bc5809c8a674-kube-api-access-jtgs6\") pod \"payload-processing-e2e-mt-65fdd4fd-77857ffd69-7jx4z\" (UID: \"73d34346-3963-480f-9081-bc5809c8a674\") " pod="openshift-ingress/payload-processing-e2e-mt-65fdd4fd-77857ffd69-7jx4z"
Sep 11 17:35:34.890827 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:34.890723    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73d34346-3963-480f-9081-bc5809c8a674-plugins-config-volume\") pod \"payload-processing-e2e-mt-65fdd4fd-77857ffd69-7jx4z\" (UID: \"73d34346-3963-480f-9081-bc5809c8a674\") " pod="openshift-ingress/payload-processing-e2e-mt-65fdd4fd-77857ffd69-7jx4z"
Sep 11 17:35:34.991873 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:34.991833    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73d34346-3963-480f-9081-bc5809c8a674-plugins-config-volume\") pod \"payload-processing-e2e-mt-65fdd4fd-77857ffd69-7jx4z\" (UID: \"73d34346-3963-480f-9081-bc5809c8a674\") " pod="openshift-ingress/payload-processing-e2e-mt-65fdd4fd-77857ffd69-7jx4z"
Sep 11 17:35:34.992078 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:34.992003    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtgs6\" (UniqueName: \"kubernetes.io/projected/73d34346-3963-480f-9081-bc5809c8a674-kube-api-access-jtgs6\") pod \"payload-processing-e2e-mt-65fdd4fd-77857ffd69-7jx4z\" (UID: \"73d34346-3963-480f-9081-bc5809c8a674\") " pod="openshift-ingress/payload-processing-e2e-mt-65fdd4fd-77857ffd69-7jx4z"
Sep 11 17:35:34.992557 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:34.992533    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73d34346-3963-480f-9081-bc5809c8a674-plugins-config-volume\") pod \"payload-processing-e2e-mt-65fdd4fd-77857ffd69-7jx4z\" (UID: \"73d34346-3963-480f-9081-bc5809c8a674\") " pod="openshift-ingress/payload-processing-e2e-mt-65fdd4fd-77857ffd69-7jx4z"
Sep 11 17:35:35.003123 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:35.003091    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtgs6\" (UniqueName: \"kubernetes.io/projected/73d34346-3963-480f-9081-bc5809c8a674-kube-api-access-jtgs6\") pod \"payload-processing-e2e-mt-65fdd4fd-77857ffd69-7jx4z\" (UID: \"73d34346-3963-480f-9081-bc5809c8a674\") " pod="openshift-ingress/payload-processing-e2e-mt-65fdd4fd-77857ffd69-7jx4z"
Sep 11 17:35:35.159737 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:35.159694    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-65fdd4fd-77857ffd69-7jx4z"
Sep 11 17:35:35.439158 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:35.439133    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-65fdd4fd-77857ffd69-7jx4z"]
Sep 11 17:35:35.440900 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:35:35.440877    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73d34346_3963_480f_9081_bc5809c8a674.slice/crio-65cbf98f5552f5968769b7b13200dfe03cacd030cb7434b15d24ed3a32bb4827 WatchSource:0}: Error finding container 65cbf98f5552f5968769b7b13200dfe03cacd030cb7434b15d24ed3a32bb4827: Status 404 returned error can't find the container with id 65cbf98f5552f5968769b7b13200dfe03cacd030cb7434b15d24ed3a32bb4827
Sep 11 17:35:35.563327 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:35.563289    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-65fdd4fd-77857ffd69-7jx4z" event={"ID":"73d34346-3963-480f-9081-bc5809c8a674","Type":"ContainerStarted","Data":"56746d09943f3c939a835dccc85e4344bbc4f9af7e1bc00d1c4a1f5aef2e35a1"}
Sep 11 17:35:35.563516 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:35.563344    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-65fdd4fd-77857ffd69-7jx4z" event={"ID":"73d34346-3963-480f-9081-bc5809c8a674","Type":"ContainerStarted","Data":"65cbf98f5552f5968769b7b13200dfe03cacd030cb7434b15d24ed3a32bb4827"}
Sep 11 17:35:35.563574 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:35.563508    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-65fdd4fd-77857ffd69-7jx4z"
Sep 11 17:35:35.581142 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:35.581095    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-65fdd4fd-77857ffd69-7jx4z" podStartSLOduration=1.581077818 podStartE2EDuration="1.581077818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:35:35.579160286 +0000 UTC m=+1015.973077744" watchObservedRunningTime="2026-09-11 17:35:35.581077818 +0000 UTC m=+1015.974995309"
Sep 11 17:35:46.570371 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:46.570340    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-65fdd4fd-77857ffd69-7jx4z"
Sep 11 17:35:59.691661 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:59.691559    2562 generic.go:356] "Generic (PLEG): container finished" podID="ea06effd-49f5-4ae9-93c8-eb1f56b13a0e" containerID="c8559e81378e89db4184ab16d081aae34ac4dbe92117a37cb265bf23352937be" exitCode=0
Sep 11 17:35:59.692320 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:59.692292    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66" event={"ID":"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e","Type":"ContainerDied","Data":"c8559e81378e89db4184ab16d081aae34ac4dbe92117a37cb265bf23352937be"}
Sep 11 17:35:59.693138 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:59.693115    2562 scope.go:117] "RemoveContainer" containerID="c8559e81378e89db4184ab16d081aae34ac4dbe92117a37cb265bf23352937be"
Sep 11 17:35:59.695831 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:59.695787    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:35:59.696214 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:59.696184    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:35:59.696382 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:35:59.696362    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:36:00.698815 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:00.698782    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66" event={"ID":"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e","Type":"ContainerStarted","Data":"b8354de973314222b6ffdd7bf10ce716e23639c5d526739fad4d06e285d80812"}
Sep 11 17:36:01.585478 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:01.585443    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"
Sep 11 17:36:01.585674 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:01.585651    2562 patch_prober.go:28] interesting pod/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 11 17:36:01.585727 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:01.585701    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66" podUID="ea06effd-49f5-4ae9-93c8-eb1f56b13a0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 11 17:36:02.585388 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:02.585341    2562 patch_prober.go:28] interesting pod/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 11 17:36:02.585846 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:02.585409    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66" podUID="ea06effd-49f5-4ae9-93c8-eb1f56b13a0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 11 17:36:03.585656 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:03.585615    2562 patch_prober.go:28] interesting pod/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 11 17:36:03.586113 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:03.585673    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66" podUID="ea06effd-49f5-4ae9-93c8-eb1f56b13a0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 11 17:36:04.585915 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:04.585875    2562 patch_prober.go:28] interesting pod/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 11 17:36:04.586362 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:04.585956    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66" podUID="ea06effd-49f5-4ae9-93c8-eb1f56b13a0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 11 17:36:04.782130 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:04.782097    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-cbbb09/test-model-cbbb09-kserve-d4dcf7cb5-mrgdn"]
Sep 11 17:36:04.787082 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:04.785861    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cbbb09/test-model-cbbb09-kserve-d4dcf7cb5-mrgdn"
Sep 11 17:36:04.789886 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:04.789856    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cbbb09\"/\"default-dockercfg-xr666\""
Sep 11 17:36:04.792051 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:04.792021    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cbbb09\"/\"kube-root-ca.crt\""
Sep 11 17:36:04.792343 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:04.792325    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cbbb09\"/\"openshift-service-ca.crt\""
Sep 11 17:36:04.792688 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:04.792586    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cbbb09\"/\"test-model-cbbb09-kserve-self-signed-certs\""
Sep 11 17:36:04.799850 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:04.799825    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cbbb09/test-model-cbbb09-kserve-d4dcf7cb5-mrgdn"]
Sep 11 17:36:04.894936 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:04.894889    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a287f58f-caa3-4b56-929b-faf88c022382-tmp-dir\") pod \"test-model-cbbb09-kserve-d4dcf7cb5-mrgdn\" (UID: \"a287f58f-caa3-4b56-929b-faf88c022382\") " pod="ai-tenant-e2e-inf-b-cbbb09/test-model-cbbb09-kserve-d4dcf7cb5-mrgdn"
Sep 11 17:36:04.895152 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:04.894950    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a287f58f-caa3-4b56-929b-faf88c022382-home\") pod \"test-model-cbbb09-kserve-d4dcf7cb5-mrgdn\" (UID: \"a287f58f-caa3-4b56-929b-faf88c022382\") " pod="ai-tenant-e2e-inf-b-cbbb09/test-model-cbbb09-kserve-d4dcf7cb5-mrgdn"
Sep 11 17:36:04.895152 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:04.894981    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a287f58f-caa3-4b56-929b-faf88c022382-tls-certs\") pod \"test-model-cbbb09-kserve-d4dcf7cb5-mrgdn\" (UID: \"a287f58f-caa3-4b56-929b-faf88c022382\") " pod="ai-tenant-e2e-inf-b-cbbb09/test-model-cbbb09-kserve-d4dcf7cb5-mrgdn"
Sep 11 17:36:04.895152 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:04.895068    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6brs4\" (UniqueName: \"kubernetes.io/projected/a287f58f-caa3-4b56-929b-faf88c022382-kube-api-access-6brs4\") pod \"test-model-cbbb09-kserve-d4dcf7cb5-mrgdn\" (UID: \"a287f58f-caa3-4b56-929b-faf88c022382\") " pod="ai-tenant-e2e-inf-b-cbbb09/test-model-cbbb09-kserve-d4dcf7cb5-mrgdn"
Sep 11 17:36:04.895152 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:04.895098    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a287f58f-caa3-4b56-929b-faf88c022382-model-cache\") pod \"test-model-cbbb09-kserve-d4dcf7cb5-mrgdn\" (UID: \"a287f58f-caa3-4b56-929b-faf88c022382\") " pod="ai-tenant-e2e-inf-b-cbbb09/test-model-cbbb09-kserve-d4dcf7cb5-mrgdn"
Sep 11 17:36:04.895372 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:04.895241    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a287f58f-caa3-4b56-929b-faf88c022382-dshm\") pod \"test-model-cbbb09-kserve-d4dcf7cb5-mrgdn\" (UID: \"a287f58f-caa3-4b56-929b-faf88c022382\") " pod="ai-tenant-e2e-inf-b-cbbb09/test-model-cbbb09-kserve-d4dcf7cb5-mrgdn"
Sep 11 17:36:04.996504 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:04.996470    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a287f58f-caa3-4b56-929b-faf88c022382-tmp-dir\") pod \"test-model-cbbb09-kserve-d4dcf7cb5-mrgdn\" (UID: \"a287f58f-caa3-4b56-929b-faf88c022382\") " pod="ai-tenant-e2e-inf-b-cbbb09/test-model-cbbb09-kserve-d4dcf7cb5-mrgdn"
Sep 11 17:36:04.996504 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:04.996510    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a287f58f-caa3-4b56-929b-faf88c022382-home\") pod \"test-model-cbbb09-kserve-d4dcf7cb5-mrgdn\" (UID: \"a287f58f-caa3-4b56-929b-faf88c022382\") " pod="ai-tenant-e2e-inf-b-cbbb09/test-model-cbbb09-kserve-d4dcf7cb5-mrgdn"
Sep 11 17:36:04.996747 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:04.996528    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a287f58f-caa3-4b56-929b-faf88c022382-tls-certs\") pod \"test-model-cbbb09-kserve-d4dcf7cb5-mrgdn\" (UID: \"a287f58f-caa3-4b56-929b-faf88c022382\") " pod="ai-tenant-e2e-inf-b-cbbb09/test-model-cbbb09-kserve-d4dcf7cb5-mrgdn"
Sep 11 17:36:04.996747 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:04.996580    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6brs4\" (UniqueName: \"kubernetes.io/projected/a287f58f-caa3-4b56-929b-faf88c022382-kube-api-access-6brs4\") pod \"test-model-cbbb09-kserve-d4dcf7cb5-mrgdn\" (UID: \"a287f58f-caa3-4b56-929b-faf88c022382\") " pod="ai-tenant-e2e-inf-b-cbbb09/test-model-cbbb09-kserve-d4dcf7cb5-mrgdn"
Sep 11 17:36:04.996747 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:04.996602    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a287f58f-caa3-4b56-929b-faf88c022382-model-cache\") pod \"test-model-cbbb09-kserve-d4dcf7cb5-mrgdn\" (UID: \"a287f58f-caa3-4b56-929b-faf88c022382\") " pod="ai-tenant-e2e-inf-b-cbbb09/test-model-cbbb09-kserve-d4dcf7cb5-mrgdn"
Sep 11 17:36:04.996909 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:04.996815    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a287f58f-caa3-4b56-929b-faf88c022382-dshm\") pod \"test-model-cbbb09-kserve-d4dcf7cb5-mrgdn\" (UID: \"a287f58f-caa3-4b56-929b-faf88c022382\") " pod="ai-tenant-e2e-inf-b-cbbb09/test-model-cbbb09-kserve-d4dcf7cb5-mrgdn"
Sep 11 17:36:04.996909 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:04.996849    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a287f58f-caa3-4b56-929b-faf88c022382-home\") pod \"test-model-cbbb09-kserve-d4dcf7cb5-mrgdn\" (UID: \"a287f58f-caa3-4b56-929b-faf88c022382\") " pod="ai-tenant-e2e-inf-b-cbbb09/test-model-cbbb09-kserve-d4dcf7cb5-mrgdn"
Sep 11 17:36:04.996909 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:04.996894    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a287f58f-caa3-4b56-929b-faf88c022382-tmp-dir\") pod \"test-model-cbbb09-kserve-d4dcf7cb5-mrgdn\" (UID: \"a287f58f-caa3-4b56-929b-faf88c022382\") " pod="ai-tenant-e2e-inf-b-cbbb09/test-model-cbbb09-kserve-d4dcf7cb5-mrgdn"
Sep 11 17:36:04.997050 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:04.996964    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a287f58f-caa3-4b56-929b-faf88c022382-model-cache\") pod \"test-model-cbbb09-kserve-d4dcf7cb5-mrgdn\" (UID: \"a287f58f-caa3-4b56-929b-faf88c022382\") " pod="ai-tenant-e2e-inf-b-cbbb09/test-model-cbbb09-kserve-d4dcf7cb5-mrgdn"
Sep 11 17:36:04.998924 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:04.998893    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a287f58f-caa3-4b56-929b-faf88c022382-dshm\") pod \"test-model-cbbb09-kserve-d4dcf7cb5-mrgdn\" (UID: \"a287f58f-caa3-4b56-929b-faf88c022382\") " pod="ai-tenant-e2e-inf-b-cbbb09/test-model-cbbb09-kserve-d4dcf7cb5-mrgdn"
Sep 11 17:36:04.999128 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:04.999110    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a287f58f-caa3-4b56-929b-faf88c022382-tls-certs\") pod \"test-model-cbbb09-kserve-d4dcf7cb5-mrgdn\" (UID: \"a287f58f-caa3-4b56-929b-faf88c022382\") " pod="ai-tenant-e2e-inf-b-cbbb09/test-model-cbbb09-kserve-d4dcf7cb5-mrgdn"
Sep 11 17:36:05.005195 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:05.005176    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6brs4\" (UniqueName: \"kubernetes.io/projected/a287f58f-caa3-4b56-929b-faf88c022382-kube-api-access-6brs4\") pod \"test-model-cbbb09-kserve-d4dcf7cb5-mrgdn\" (UID: \"a287f58f-caa3-4b56-929b-faf88c022382\") " pod="ai-tenant-e2e-inf-b-cbbb09/test-model-cbbb09-kserve-d4dcf7cb5-mrgdn"
Sep 11 17:36:05.101421 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:05.101383    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cbbb09/test-model-cbbb09-kserve-d4dcf7cb5-mrgdn"
Sep 11 17:36:05.394795 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:05.394769    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cbbb09/test-model-cbbb09-kserve-d4dcf7cb5-mrgdn"]
Sep 11 17:36:05.396473 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:36:05.396446    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda287f58f_caa3_4b56_929b_faf88c022382.slice/crio-e5c4e9f90c0a87341aad065bd5a92768001ec1872ca310d2a508af9439f719df WatchSource:0}: Error finding container e5c4e9f90c0a87341aad065bd5a92768001ec1872ca310d2a508af9439f719df: Status 404 returned error can't find the container with id e5c4e9f90c0a87341aad065bd5a92768001ec1872ca310d2a508af9439f719df
Sep 11 17:36:05.585388 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:05.585349    2562 patch_prober.go:28] interesting pod/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 11 17:36:05.585585 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:05.585414    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66" podUID="ea06effd-49f5-4ae9-93c8-eb1f56b13a0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 11 17:36:05.726433 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:05.726330    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cbbb09/test-model-cbbb09-kserve-d4dcf7cb5-mrgdn" event={"ID":"a287f58f-caa3-4b56-929b-faf88c022382","Type":"ContainerStarted","Data":"34f8492f07d51c5dd15c27df537a758437ee7bf4b61f161ad924c06cc776daa4"}
Sep 11 17:36:05.726433 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:05.726379    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cbbb09/test-model-cbbb09-kserve-d4dcf7cb5-mrgdn" event={"ID":"a287f58f-caa3-4b56-929b-faf88c022382","Type":"ContainerStarted","Data":"e5c4e9f90c0a87341aad065bd5a92768001ec1872ca310d2a508af9439f719df"}
Sep 11 17:36:05.751022 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:05.748670    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-cbbb09/test-model-cbbb09-kserve-d4dcf7cb5-mrgdn" podStartSLOduration=1.748647877 podStartE2EDuration="1.748647877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:36:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:36:05.747630172 +0000 UTC m=+1046.141547700" watchObservedRunningTime="2026-09-11 17:36:05.748647877 +0000 UTC m=+1046.142565333"
Sep 11 17:36:06.589084 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:06.589052    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"
Sep 11 17:36:06.589394 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:06.589375    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"
Sep 11 17:36:06.590084 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:06.590065    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"
Sep 11 17:36:15.102395 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:15.102359    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-cbbb09/test-model-cbbb09-kserve-d4dcf7cb5-mrgdn"
Sep 11 17:36:15.102784 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:15.102410    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-cbbb09/test-model-cbbb09-kserve-d4dcf7cb5-mrgdn"
Sep 11 17:36:15.114979 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:15.114953    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-cbbb09/test-model-cbbb09-kserve-d4dcf7cb5-mrgdn"
Sep 11 17:36:15.786005 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:15.785964    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-cbbb09/test-model-cbbb09-kserve-d4dcf7cb5-mrgdn"
Sep 11 17:36:17.954196 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:17.954108    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-65fdd4fd-77857ffd69-7jx4z"]
Sep 11 17:36:17.954630 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:17.954317    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-65fdd4fd-77857ffd69-7jx4z" podUID="73d34346-3963-480f-9081-bc5809c8a674" containerName="payload-processing" containerID="cri-o://56746d09943f3c939a835dccc85e4344bbc4f9af7e1bc00d1c4a1f5aef2e35a1" gracePeriod=30
Sep 11 17:36:18.219518 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:18.219431    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-65fdd4fd-77857ffd69-7jx4z"
Sep 11 17:36:18.332466 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:18.332432    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73d34346-3963-480f-9081-bc5809c8a674-plugins-config-volume\") pod \"73d34346-3963-480f-9081-bc5809c8a674\" (UID: \"73d34346-3963-480f-9081-bc5809c8a674\") "
Sep 11 17:36:18.332685 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:18.332566    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtgs6\" (UniqueName: \"kubernetes.io/projected/73d34346-3963-480f-9081-bc5809c8a674-kube-api-access-jtgs6\") pod \"73d34346-3963-480f-9081-bc5809c8a674\" (UID: \"73d34346-3963-480f-9081-bc5809c8a674\") "
Sep 11 17:36:18.332931 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:18.332889    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73d34346-3963-480f-9081-bc5809c8a674-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "73d34346-3963-480f-9081-bc5809c8a674" (UID: "73d34346-3963-480f-9081-bc5809c8a674"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:36:18.334745 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:18.334707    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73d34346-3963-480f-9081-bc5809c8a674-kube-api-access-jtgs6" (OuterVolumeSpecName: "kube-api-access-jtgs6") pod "73d34346-3963-480f-9081-bc5809c8a674" (UID: "73d34346-3963-480f-9081-bc5809c8a674"). InnerVolumeSpecName "kube-api-access-jtgs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:36:18.434184 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:18.434150    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtgs6\" (UniqueName: \"kubernetes.io/projected/73d34346-3963-480f-9081-bc5809c8a674-kube-api-access-jtgs6\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:36:18.434184 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:18.434179    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73d34346-3963-480f-9081-bc5809c8a674-plugins-config-volume\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:36:18.794192 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:18.794078    2562 generic.go:356] "Generic (PLEG): container finished" podID="73d34346-3963-480f-9081-bc5809c8a674" containerID="56746d09943f3c939a835dccc85e4344bbc4f9af7e1bc00d1c4a1f5aef2e35a1" exitCode=0
Sep 11 17:36:18.794192 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:18.794121    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-65fdd4fd-77857ffd69-7jx4z" event={"ID":"73d34346-3963-480f-9081-bc5809c8a674","Type":"ContainerDied","Data":"56746d09943f3c939a835dccc85e4344bbc4f9af7e1bc00d1c4a1f5aef2e35a1"}
Sep 11 17:36:18.794192 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:18.794191    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-65fdd4fd-77857ffd69-7jx4z" event={"ID":"73d34346-3963-480f-9081-bc5809c8a674","Type":"ContainerDied","Data":"65cbf98f5552f5968769b7b13200dfe03cacd030cb7434b15d24ed3a32bb4827"}
Sep 11 17:36:18.794560 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:18.794221    2562 scope.go:117] "RemoveContainer" containerID="56746d09943f3c939a835dccc85e4344bbc4f9af7e1bc00d1c4a1f5aef2e35a1"
Sep 11 17:36:18.795150 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:18.794933    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-65fdd4fd-77857ffd69-7jx4z"
Sep 11 17:36:18.804533 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:18.804513    2562 scope.go:117] "RemoveContainer" containerID="56746d09943f3c939a835dccc85e4344bbc4f9af7e1bc00d1c4a1f5aef2e35a1"
Sep 11 17:36:18.804844 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:36:18.804824    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56746d09943f3c939a835dccc85e4344bbc4f9af7e1bc00d1c4a1f5aef2e35a1\": container with ID starting with 56746d09943f3c939a835dccc85e4344bbc4f9af7e1bc00d1c4a1f5aef2e35a1 not found: ID does not exist" containerID="56746d09943f3c939a835dccc85e4344bbc4f9af7e1bc00d1c4a1f5aef2e35a1"
Sep 11 17:36:18.804896 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:18.804855    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56746d09943f3c939a835dccc85e4344bbc4f9af7e1bc00d1c4a1f5aef2e35a1"} err="failed to get container status \"56746d09943f3c939a835dccc85e4344bbc4f9af7e1bc00d1c4a1f5aef2e35a1\": rpc error: code = NotFound desc = could not find container \"56746d09943f3c939a835dccc85e4344bbc4f9af7e1bc00d1c4a1f5aef2e35a1\": container with ID starting with 56746d09943f3c939a835dccc85e4344bbc4f9af7e1bc00d1c4a1f5aef2e35a1 not found: ID does not exist"
Sep 11 17:36:18.822885 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:18.822862    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-65fdd4fd-77857ffd69-7jx4z"]
Sep 11 17:36:18.829264 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:18.829242    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-65fdd4fd-77857ffd69-7jx4z"]
Sep 11 17:36:20.126412 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:20.126381    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73d34346-3963-480f-9081-bc5809c8a674" path="/var/lib/kubelet/pods/73d34346-3963-480f-9081-bc5809c8a674/volumes"
Sep 11 17:36:25.127497 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:25.127466    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"]
Sep 11 17:36:25.127972 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:25.127730    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66" podUID="ea06effd-49f5-4ae9-93c8-eb1f56b13a0e" containerName="istio-proxy" containerID="cri-o://b8354de973314222b6ffdd7bf10ce716e23639c5d526739fad4d06e285d80812" gracePeriod=30
Sep 11 17:36:30.392413 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.392354    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"
Sep 11 17:36:30.445179 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.445144    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-workload-certs\") pod \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\" (UID: \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\") "
Sep 11 17:36:30.445179 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.445182    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8klb\" (UniqueName: \"kubernetes.io/projected/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-kube-api-access-z8klb\") pod \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\" (UID: \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\") "
Sep 11 17:36:30.445387 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.445231    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-istiod-ca-cert\") pod \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\" (UID: \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\") "
Sep 11 17:36:30.445387 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.445261    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-credential-socket\") pod \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\" (UID: \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\") "
Sep 11 17:36:30.445387 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.445323    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-istio-podinfo\") pod \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\" (UID: \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\") "
Sep 11 17:36:30.445387 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.445355    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-workload-socket\") pod \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\" (UID: \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\") "
Sep 11 17:36:30.445387 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.445379    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-istio-data\") pod \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\" (UID: \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\") "
Sep 11 17:36:30.445641 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.445410    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-istio-token\") pod \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\" (UID: \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\") "
Sep 11 17:36:30.445641 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.445456    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ea06effd-49f5-4ae9-93c8-eb1f56b13a0e" (UID: "ea06effd-49f5-4ae9-93c8-eb1f56b13a0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:36:30.445641 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.445472    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-istio-envoy\") pod \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\" (UID: \"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e\") "
Sep 11 17:36:30.445803 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.445668    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ea06effd-49f5-4ae9-93c8-eb1f56b13a0e" (UID: "ea06effd-49f5-4ae9-93c8-eb1f56b13a0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:36:30.445803 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.445709    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ea06effd-49f5-4ae9-93c8-eb1f56b13a0e" (UID: "ea06effd-49f5-4ae9-93c8-eb1f56b13a0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:36:30.445908 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.445874    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-istio-data" (OuterVolumeSpecName: "istio-data") pod "ea06effd-49f5-4ae9-93c8-eb1f56b13a0e" (UID: "ea06effd-49f5-4ae9-93c8-eb1f56b13a0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:36:30.445964 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.445927    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-workload-certs\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:36:30.445964 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.445949    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-credential-socket\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:36:30.446105 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.445963    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-workload-socket\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:36:30.446168 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.446093    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ea06effd-49f5-4ae9-93c8-eb1f56b13a0e" (UID: "ea06effd-49f5-4ae9-93c8-eb1f56b13a0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:36:30.447590 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.447562    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ea06effd-49f5-4ae9-93c8-eb1f56b13a0e" (UID: "ea06effd-49f5-4ae9-93c8-eb1f56b13a0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:36:30.447712 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.447622    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ea06effd-49f5-4ae9-93c8-eb1f56b13a0e" (UID: "ea06effd-49f5-4ae9-93c8-eb1f56b13a0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 17:36:30.447816 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.447798    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-kube-api-access-z8klb" (OuterVolumeSpecName: "kube-api-access-z8klb") pod "ea06effd-49f5-4ae9-93c8-eb1f56b13a0e" (UID: "ea06effd-49f5-4ae9-93c8-eb1f56b13a0e"). InnerVolumeSpecName "kube-api-access-z8klb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:36:30.447954 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.447931    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-istio-token" (OuterVolumeSpecName: "istio-token") pod "ea06effd-49f5-4ae9-93c8-eb1f56b13a0e" (UID: "ea06effd-49f5-4ae9-93c8-eb1f56b13a0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:36:30.547240 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.547202    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-istio-podinfo\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:36:30.547240 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.547236    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-istio-data\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:36:30.547240 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.547246    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-istio-token\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:36:30.547471 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.547253    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-istio-envoy\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:36:30.547471 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.547262    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8klb\" (UniqueName: \"kubernetes.io/projected/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-kube-api-access-z8klb\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:36:30.547471 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.547271    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e-istiod-ca-cert\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:36:30.857673 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.857641    2562 generic.go:356] "Generic (PLEG): container finished" podID="ea06effd-49f5-4ae9-93c8-eb1f56b13a0e" containerID="b8354de973314222b6ffdd7bf10ce716e23639c5d526739fad4d06e285d80812" exitCode=0
Sep 11 17:36:30.857844 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.857695    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"
Sep 11 17:36:30.857844 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.857709    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66" event={"ID":"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e","Type":"ContainerDied","Data":"b8354de973314222b6ffdd7bf10ce716e23639c5d526739fad4d06e285d80812"}
Sep 11 17:36:30.857844 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.857804    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66" event={"ID":"ea06effd-49f5-4ae9-93c8-eb1f56b13a0e","Type":"ContainerDied","Data":"c94446b9ab0d6dcb6531b551f4148eb240ce61ad2e8b04c345a0c77d257667ae"}
Sep 11 17:36:30.857844 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.857836    2562 scope.go:117] "RemoveContainer" containerID="b8354de973314222b6ffdd7bf10ce716e23639c5d526739fad4d06e285d80812"
Sep 11 17:36:30.867385 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.867365    2562 scope.go:117] "RemoveContainer" containerID="c8559e81378e89db4184ab16d081aae34ac4dbe92117a37cb265bf23352937be"
Sep 11 17:36:30.874948 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.874927    2562 scope.go:117] "RemoveContainer" containerID="b8354de973314222b6ffdd7bf10ce716e23639c5d526739fad4d06e285d80812"
Sep 11 17:36:30.875250 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:36:30.875227    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8354de973314222b6ffdd7bf10ce716e23639c5d526739fad4d06e285d80812\": container with ID starting with b8354de973314222b6ffdd7bf10ce716e23639c5d526739fad4d06e285d80812 not found: ID does not exist" containerID="b8354de973314222b6ffdd7bf10ce716e23639c5d526739fad4d06e285d80812"
Sep 11 17:36:30.875327 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.875260    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8354de973314222b6ffdd7bf10ce716e23639c5d526739fad4d06e285d80812"} err="failed to get container status \"b8354de973314222b6ffdd7bf10ce716e23639c5d526739fad4d06e285d80812\": rpc error: code = NotFound desc = could not find container \"b8354de973314222b6ffdd7bf10ce716e23639c5d526739fad4d06e285d80812\": container with ID starting with b8354de973314222b6ffdd7bf10ce716e23639c5d526739fad4d06e285d80812 not found: ID does not exist"
Sep 11 17:36:30.875327 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.875284    2562 scope.go:117] "RemoveContainer" containerID="c8559e81378e89db4184ab16d081aae34ac4dbe92117a37cb265bf23352937be"
Sep 11 17:36:30.875580 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:36:30.875559    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8559e81378e89db4184ab16d081aae34ac4dbe92117a37cb265bf23352937be\": container with ID starting with c8559e81378e89db4184ab16d081aae34ac4dbe92117a37cb265bf23352937be not found: ID does not exist" containerID="c8559e81378e89db4184ab16d081aae34ac4dbe92117a37cb265bf23352937be"
Sep 11 17:36:30.875634 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.875590    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8559e81378e89db4184ab16d081aae34ac4dbe92117a37cb265bf23352937be"} err="failed to get container status \"c8559e81378e89db4184ab16d081aae34ac4dbe92117a37cb265bf23352937be\": rpc error: code = NotFound desc = could not find container \"c8559e81378e89db4184ab16d081aae34ac4dbe92117a37cb265bf23352937be\": container with ID starting with c8559e81378e89db4184ab16d081aae34ac4dbe92117a37cb265bf23352937be not found: ID does not exist"
Sep 11 17:36:30.907660 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.907633    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"]
Sep 11 17:36:30.912357 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:30.912330    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-65fdd4fd-openshift-default-6667fbb5b8-hqb66"]
Sep 11 17:36:32.126597 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:32.126521    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea06effd-49f5-4ae9-93c8-eb1f56b13a0e" path="/var/lib/kubelet/pods/ea06effd-49f5-4ae9-93c8-eb1f56b13a0e/volumes"
Sep 11 17:36:47.703887 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:47.703848    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-290683-7546dcdd98-pnm98"]
Sep 11 17:36:47.704313 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:47.704238    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea06effd-49f5-4ae9-93c8-eb1f56b13a0e" containerName="istio-proxy"
Sep 11 17:36:47.704313 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:47.704251    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea06effd-49f5-4ae9-93c8-eb1f56b13a0e" containerName="istio-proxy"
Sep 11 17:36:47.704313 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:47.704271    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73d34346-3963-480f-9081-bc5809c8a674" containerName="payload-processing"
Sep 11 17:36:47.704313 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:47.704277    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="73d34346-3963-480f-9081-bc5809c8a674" containerName="payload-processing"
Sep 11 17:36:47.704465 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:47.704334    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea06effd-49f5-4ae9-93c8-eb1f56b13a0e" containerName="istio-proxy"
Sep 11 17:36:47.704465 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:47.704343    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea06effd-49f5-4ae9-93c8-eb1f56b13a0e" containerName="istio-proxy"
Sep 11 17:36:47.704465 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:47.704352    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="73d34346-3963-480f-9081-bc5809c8a674" containerName="payload-processing"
Sep 11 17:36:47.707320 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:47.707303    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-290683-7546dcdd98-pnm98"
Sep 11 17:36:47.711433 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:47.711411    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-290683-dockercfg-4j6x5\""
Sep 11 17:36:47.712113 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:47.712095    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-290683\""
Sep 11 17:36:47.721976 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:47.721951    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-290683-7546dcdd98-pnm98"]
Sep 11 17:36:47.800668 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:47.800637    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-290683-555989d749-5zcfd"]
Sep 11 17:36:47.801046 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:47.801032    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea06effd-49f5-4ae9-93c8-eb1f56b13a0e" containerName="istio-proxy"
Sep 11 17:36:47.801046 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:47.801047    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea06effd-49f5-4ae9-93c8-eb1f56b13a0e" containerName="istio-proxy"
Sep 11 17:36:47.803370 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:47.803342    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-290683-555989d749-5zcfd"
Sep 11 17:36:47.819233 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:47.819202    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-290683-555989d749-5zcfd"]
Sep 11 17:36:47.821980 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:47.821954    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8021f9fd-66b1-40ea-b776-2119913955d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-290683-7546dcdd98-pnm98\" (UID: \"8021f9fd-66b1-40ea-b776-2119913955d6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-290683-7546dcdd98-pnm98"
Sep 11 17:36:47.822115 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:47.822028    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcldd\" (UniqueName: \"kubernetes.io/projected/8021f9fd-66b1-40ea-b776-2119913955d6-kube-api-access-lcldd\") pod \"payload-pre-processing-e2e-shared-a-w2-290683-7546dcdd98-pnm98\" (UID: \"8021f9fd-66b1-40ea-b776-2119913955d6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-290683-7546dcdd98-pnm98"
Sep 11 17:36:47.923332 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:47.923297    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8021f9fd-66b1-40ea-b776-2119913955d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-290683-7546dcdd98-pnm98\" (UID: \"8021f9fd-66b1-40ea-b776-2119913955d6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-290683-7546dcdd98-pnm98"
Sep 11 17:36:47.923498 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:47.923363    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcldd\" (UniqueName: \"kubernetes.io/projected/8021f9fd-66b1-40ea-b776-2119913955d6-kube-api-access-lcldd\") pod \"payload-pre-processing-e2e-shared-a-w2-290683-7546dcdd98-pnm98\" (UID: \"8021f9fd-66b1-40ea-b776-2119913955d6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-290683-7546dcdd98-pnm98"
Sep 11 17:36:47.923498 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:47.923452    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrl46\" (UniqueName: \"kubernetes.io/projected/05f0381c-b8e4-46d3-9046-888477b55704-kube-api-access-vrl46\") pod \"payload-processing-e2e-shared-a-w2-290683-555989d749-5zcfd\" (UID: \"05f0381c-b8e4-46d3-9046-888477b55704\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-290683-555989d749-5zcfd"
Sep 11 17:36:47.923583 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:47.923511    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05f0381c-b8e4-46d3-9046-888477b55704-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-290683-555989d749-5zcfd\" (UID: \"05f0381c-b8e4-46d3-9046-888477b55704\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-290683-555989d749-5zcfd"
Sep 11 17:36:47.924027 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:47.924008    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8021f9fd-66b1-40ea-b776-2119913955d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-290683-7546dcdd98-pnm98\" (UID: \"8021f9fd-66b1-40ea-b776-2119913955d6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-290683-7546dcdd98-pnm98"
Sep 11 17:36:47.932804 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:47.932781    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcldd\" (UniqueName: \"kubernetes.io/projected/8021f9fd-66b1-40ea-b776-2119913955d6-kube-api-access-lcldd\") pod \"payload-pre-processing-e2e-shared-a-w2-290683-7546dcdd98-pnm98\" (UID: \"8021f9fd-66b1-40ea-b776-2119913955d6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-290683-7546dcdd98-pnm98"
Sep 11 17:36:48.017257 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:48.017168    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-290683-7546dcdd98-pnm98"
Sep 11 17:36:48.025582 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:48.025540    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrl46\" (UniqueName: \"kubernetes.io/projected/05f0381c-b8e4-46d3-9046-888477b55704-kube-api-access-vrl46\") pod \"payload-processing-e2e-shared-a-w2-290683-555989d749-5zcfd\" (UID: \"05f0381c-b8e4-46d3-9046-888477b55704\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-290683-555989d749-5zcfd"
Sep 11 17:36:48.025724 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:48.025625    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05f0381c-b8e4-46d3-9046-888477b55704-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-290683-555989d749-5zcfd\" (UID: \"05f0381c-b8e4-46d3-9046-888477b55704\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-290683-555989d749-5zcfd"
Sep 11 17:36:48.026488 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:48.026460    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05f0381c-b8e4-46d3-9046-888477b55704-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-290683-555989d749-5zcfd\" (UID: \"05f0381c-b8e4-46d3-9046-888477b55704\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-290683-555989d749-5zcfd"
Sep 11 17:36:48.041521 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:48.041490    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrl46\" (UniqueName: \"kubernetes.io/projected/05f0381c-b8e4-46d3-9046-888477b55704-kube-api-access-vrl46\") pod \"payload-processing-e2e-shared-a-w2-290683-555989d749-5zcfd\" (UID: \"05f0381c-b8e4-46d3-9046-888477b55704\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-290683-555989d749-5zcfd"
Sep 11 17:36:48.116308 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:48.116278    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-290683-555989d749-5zcfd"
Sep 11 17:36:48.305194 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:48.305162    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-290683-7546dcdd98-pnm98"]
Sep 11 17:36:48.307148 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:36:48.307125    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8021f9fd_66b1_40ea_b776_2119913955d6.slice/crio-1075f5a66d1841de6cbe8180b580b9b8813978e16e1ba8ec397c00c033fed6be WatchSource:0}: Error finding container 1075f5a66d1841de6cbe8180b580b9b8813978e16e1ba8ec397c00c033fed6be: Status 404 returned error can't find the container with id 1075f5a66d1841de6cbe8180b580b9b8813978e16e1ba8ec397c00c033fed6be
Sep 11 17:36:48.400504 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:48.400481    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-290683-555989d749-5zcfd"]
Sep 11 17:36:48.401823 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:36:48.401800    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f0381c_b8e4_46d3_9046_888477b55704.slice/crio-2fdd448870b3e718ddfe2be9e6c448687d6d41255c120f228e16886d2be70d53 WatchSource:0}: Error finding container 2fdd448870b3e718ddfe2be9e6c448687d6d41255c120f228e16886d2be70d53: Status 404 returned error can't find the container with id 2fdd448870b3e718ddfe2be9e6c448687d6d41255c120f228e16886d2be70d53
Sep 11 17:36:48.946961 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:48.946922    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-290683-555989d749-5zcfd" event={"ID":"05f0381c-b8e4-46d3-9046-888477b55704","Type":"ContainerStarted","Data":"c06363fc49987cd3ac3797bb3b5be91cafc651c87785c8da2ec1d12d8f82a8e2"}
Sep 11 17:36:48.947452 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:48.947049    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-290683-555989d749-5zcfd"
Sep 11 17:36:48.947452 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:48.947082    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-290683-555989d749-5zcfd" event={"ID":"05f0381c-b8e4-46d3-9046-888477b55704","Type":"ContainerStarted","Data":"2fdd448870b3e718ddfe2be9e6c448687d6d41255c120f228e16886d2be70d53"}
Sep 11 17:36:48.948922 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:48.948897    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-290683-7546dcdd98-pnm98" event={"ID":"8021f9fd-66b1-40ea-b776-2119913955d6","Type":"ContainerStarted","Data":"c92cc67474c8ec84b5f8dbb971402f27617b6f5ec6b1c67d8308a6c4e55bcd13"}
Sep 11 17:36:48.949076 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:48.948946    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-290683-7546dcdd98-pnm98"
Sep 11 17:36:48.949076 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:48.948964    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-290683-7546dcdd98-pnm98" event={"ID":"8021f9fd-66b1-40ea-b776-2119913955d6","Type":"ContainerStarted","Data":"1075f5a66d1841de6cbe8180b580b9b8813978e16e1ba8ec397c00c033fed6be"}
Sep 11 17:36:48.970413 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:48.970354    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-290683-555989d749-5zcfd" podStartSLOduration=1.970339451 podStartE2EDuration="1.970339451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:36:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:36:48.967983377 +0000 UTC m=+1089.361900834" watchObservedRunningTime="2026-09-11 17:36:48.970339451 +0000 UTC m=+1089.364256905"
Sep 11 17:36:48.997858 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:48.997810    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-290683-7546dcdd98-pnm98" podStartSLOduration=1.997796207 podStartE2EDuration="1.997796207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:36:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:36:48.995792922 +0000 UTC m=+1089.389710377" watchObservedRunningTime="2026-09-11 17:36:48.997796207 +0000 UTC m=+1089.391713661"
Sep 11 17:36:59.955833 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:59.955799    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-290683-555989d749-5zcfd"
Sep 11 17:36:59.956320 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:36:59.955965    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-290683-7546dcdd98-pnm98"
Sep 11 17:37:23.948293 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:23.948259    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2f0a78-765f68f474-88w8m"]
Sep 11 17:37:23.953140 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:23.953115    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2f0a78-765f68f474-88w8m"
Sep 11 17:37:23.957836 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:23.957807    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-2f0a78-dockercfg-qfstq\""
Sep 11 17:37:23.961245 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:23.961221    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-2f0a78\""
Sep 11 17:37:23.983563 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:23.983535    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-2f0a78-59bd4ff499-qcqkn"]
Sep 11 17:37:23.988950 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:23.988929    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2f0a78-59bd4ff499-qcqkn"
Sep 11 17:37:24.019408 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:24.019366    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-2f0a78-59bd4ff499-qcqkn"]
Sep 11 17:37:24.039435 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:24.039400    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2f0a78-765f68f474-88w8m"]
Sep 11 17:37:24.069665 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:24.069594    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39cd2308-d436-41ca-b9e8-02078533374e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-2f0a78-59bd4ff499-qcqkn\" (UID: \"39cd2308-d436-41ca-b9e8-02078533374e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2f0a78-59bd4ff499-qcqkn"
Sep 11 17:37:24.069665 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:24.069676    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f64a7c90-1859-4bb7-afa8-76b8c51f2527-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-2f0a78-765f68f474-88w8m\" (UID: \"f64a7c90-1859-4bb7-afa8-76b8c51f2527\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2f0a78-765f68f474-88w8m"
Sep 11 17:37:24.069939 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:24.069770    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4bjt\" (UniqueName: \"kubernetes.io/projected/39cd2308-d436-41ca-b9e8-02078533374e-kube-api-access-s4bjt\") pod \"payload-processing-e2e-shared-b-w2-2f0a78-59bd4ff499-qcqkn\" (UID: \"39cd2308-d436-41ca-b9e8-02078533374e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2f0a78-59bd4ff499-qcqkn"
Sep 11 17:37:24.069939 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:24.069821    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wch29\" (UniqueName: \"kubernetes.io/projected/f64a7c90-1859-4bb7-afa8-76b8c51f2527-kube-api-access-wch29\") pod \"payload-pre-processing-e2e-shared-b-w2-2f0a78-765f68f474-88w8m\" (UID: \"f64a7c90-1859-4bb7-afa8-76b8c51f2527\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2f0a78-765f68f474-88w8m"
Sep 11 17:37:24.170415 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:24.170383    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39cd2308-d436-41ca-b9e8-02078533374e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-2f0a78-59bd4ff499-qcqkn\" (UID: \"39cd2308-d436-41ca-b9e8-02078533374e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2f0a78-59bd4ff499-qcqkn"
Sep 11 17:37:24.170598 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:24.170434    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f64a7c90-1859-4bb7-afa8-76b8c51f2527-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-2f0a78-765f68f474-88w8m\" (UID: \"f64a7c90-1859-4bb7-afa8-76b8c51f2527\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2f0a78-765f68f474-88w8m"
Sep 11 17:37:24.170598 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:24.170462    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4bjt\" (UniqueName: \"kubernetes.io/projected/39cd2308-d436-41ca-b9e8-02078533374e-kube-api-access-s4bjt\") pod \"payload-processing-e2e-shared-b-w2-2f0a78-59bd4ff499-qcqkn\" (UID: \"39cd2308-d436-41ca-b9e8-02078533374e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2f0a78-59bd4ff499-qcqkn"
Sep 11 17:37:24.170598 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:24.170496    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wch29\" (UniqueName: \"kubernetes.io/projected/f64a7c90-1859-4bb7-afa8-76b8c51f2527-kube-api-access-wch29\") pod \"payload-pre-processing-e2e-shared-b-w2-2f0a78-765f68f474-88w8m\" (UID: \"f64a7c90-1859-4bb7-afa8-76b8c51f2527\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2f0a78-765f68f474-88w8m"
Sep 11 17:37:24.171421 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:24.171394    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f64a7c90-1859-4bb7-afa8-76b8c51f2527-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-2f0a78-765f68f474-88w8m\" (UID: \"f64a7c90-1859-4bb7-afa8-76b8c51f2527\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2f0a78-765f68f474-88w8m"
Sep 11 17:37:24.171496 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:24.171394    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39cd2308-d436-41ca-b9e8-02078533374e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-2f0a78-59bd4ff499-qcqkn\" (UID: \"39cd2308-d436-41ca-b9e8-02078533374e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2f0a78-59bd4ff499-qcqkn"
Sep 11 17:37:24.180611 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:24.180580    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4bjt\" (UniqueName: \"kubernetes.io/projected/39cd2308-d436-41ca-b9e8-02078533374e-kube-api-access-s4bjt\") pod \"payload-processing-e2e-shared-b-w2-2f0a78-59bd4ff499-qcqkn\" (UID: \"39cd2308-d436-41ca-b9e8-02078533374e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2f0a78-59bd4ff499-qcqkn"
Sep 11 17:37:24.181215 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:24.181192    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wch29\" (UniqueName: \"kubernetes.io/projected/f64a7c90-1859-4bb7-afa8-76b8c51f2527-kube-api-access-wch29\") pod \"payload-pre-processing-e2e-shared-b-w2-2f0a78-765f68f474-88w8m\" (UID: \"f64a7c90-1859-4bb7-afa8-76b8c51f2527\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2f0a78-765f68f474-88w8m"
Sep 11 17:37:24.264521 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:24.264439    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2f0a78-765f68f474-88w8m"
Sep 11 17:37:24.298898 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:24.298864    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2f0a78-59bd4ff499-qcqkn"
Sep 11 17:37:24.549751 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:24.549705    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2f0a78-765f68f474-88w8m"]
Sep 11 17:37:24.552921 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:37:24.552894    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf64a7c90_1859_4bb7_afa8_76b8c51f2527.slice/crio-392c129751cc0c3a4d1bda402d3ead615497efc5aff2fba595ed2d6b703a79c1 WatchSource:0}: Error finding container 392c129751cc0c3a4d1bda402d3ead615497efc5aff2fba595ed2d6b703a79c1: Status 404 returned error can't find the container with id 392c129751cc0c3a4d1bda402d3ead615497efc5aff2fba595ed2d6b703a79c1
Sep 11 17:37:24.583252 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:24.583227    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-2f0a78-59bd4ff499-qcqkn"]
Sep 11 17:37:24.587127 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:37:24.587098    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39cd2308_d436_41ca_b9e8_02078533374e.slice/crio-1e45a9132807ce657195840016e48266519af65fe6cf63191afced5c2e5138f1 WatchSource:0}: Error finding container 1e45a9132807ce657195840016e48266519af65fe6cf63191afced5c2e5138f1: Status 404 returned error can't find the container with id 1e45a9132807ce657195840016e48266519af65fe6cf63191afced5c2e5138f1
Sep 11 17:37:25.135653 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:25.135618    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2f0a78-59bd4ff499-qcqkn" event={"ID":"39cd2308-d436-41ca-b9e8-02078533374e","Type":"ContainerStarted","Data":"13a22d95b8fcfedd976f6f420fc73cad833e8ea2fe1f8cf7dbca0137b5c50d16"}
Sep 11 17:37:25.136169 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:25.135743    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2f0a78-59bd4ff499-qcqkn" event={"ID":"39cd2308-d436-41ca-b9e8-02078533374e","Type":"ContainerStarted","Data":"1e45a9132807ce657195840016e48266519af65fe6cf63191afced5c2e5138f1"}
Sep 11 17:37:25.136169 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:25.135784    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2f0a78-59bd4ff499-qcqkn"
Sep 11 17:37:25.137419 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:25.137389    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2f0a78-765f68f474-88w8m" event={"ID":"f64a7c90-1859-4bb7-afa8-76b8c51f2527","Type":"ContainerStarted","Data":"881ca0fcb951782a8282ae59e2d983ac509b92287122e1497b52423162098a4a"}
Sep 11 17:37:25.137554 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:25.137437    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2f0a78-765f68f474-88w8m" event={"ID":"f64a7c90-1859-4bb7-afa8-76b8c51f2527","Type":"ContainerStarted","Data":"392c129751cc0c3a4d1bda402d3ead615497efc5aff2fba595ed2d6b703a79c1"}
Sep 11 17:37:25.137554 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:25.137472    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2f0a78-765f68f474-88w8m"
Sep 11 17:37:25.164788 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:25.164730    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2f0a78-59bd4ff499-qcqkn" podStartSLOduration=2.16471415 podStartE2EDuration="2.16471415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:37:25.161832916 +0000 UTC m=+1125.555750370" watchObservedRunningTime="2026-09-11 17:37:25.16471415 +0000 UTC m=+1125.558631655"
Sep 11 17:37:25.199554 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:25.199498    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2f0a78-765f68f474-88w8m" podStartSLOduration=2.19948119 podStartE2EDuration="2.19948119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:37:25.198112059 +0000 UTC m=+1125.592029513" watchObservedRunningTime="2026-09-11 17:37:25.19948119 +0000 UTC m=+1125.593398643"
Sep 11 17:37:36.144384 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:36.144353    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2f0a78-59bd4ff499-qcqkn"
Sep 11 17:37:36.144762 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:36.144428    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2f0a78-765f68f474-88w8m"
Sep 11 17:37:54.763699 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:54.763666    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ec3fed-7865cdbb49-4dpfb"]
Sep 11 17:37:54.764362 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:54.763858    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-ec3fed-7865cdbb49-4dpfb" podUID="31921d54-4a87-46f4-aed5-4954c6946bd3" containerName="payload-processing" containerID="cri-o://bf17fbcf2911bc01d8dfe9ff3696d3b5d24dc3c5c2492ec4191a5f2b13c7fea4" gracePeriod=30
Sep 11 17:37:54.785043 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:54.785015    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ec3fed-567c4fd854-rt5x7"]
Sep 11 17:37:54.785280 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:54.785239    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ec3fed-567c4fd854-rt5x7" podUID="9ffb5bd3-cb59-415a-b061-6084950220de" containerName="payload-pre-processing" containerID="cri-o://510aa455c9bd7d8c1199b82a8d8b55eeff2f396060775118f139f02e283e8839" gracePeriod=30
Sep 11 17:37:55.062116 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:55.062088    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-ec3fed-7865cdbb49-4dpfb"
Sep 11 17:37:55.065146 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:55.065126    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ec3fed-567c4fd854-rt5x7"
Sep 11 17:37:55.171774 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:55.171739    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31921d54-4a87-46f4-aed5-4954c6946bd3-plugins-config-volume\") pod \"31921d54-4a87-46f4-aed5-4954c6946bd3\" (UID: \"31921d54-4a87-46f4-aed5-4954c6946bd3\") "
Sep 11 17:37:55.171971 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:55.171788    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ffb5bd3-cb59-415a-b061-6084950220de-plugins-config-volume\") pod \"9ffb5bd3-cb59-415a-b061-6084950220de\" (UID: \"9ffb5bd3-cb59-415a-b061-6084950220de\") "
Sep 11 17:37:55.171971 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:55.171852    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h46wb\" (UniqueName: \"kubernetes.io/projected/31921d54-4a87-46f4-aed5-4954c6946bd3-kube-api-access-h46wb\") pod \"31921d54-4a87-46f4-aed5-4954c6946bd3\" (UID: \"31921d54-4a87-46f4-aed5-4954c6946bd3\") "
Sep 11 17:37:55.171971 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:55.171902    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfbn4\" (UniqueName: \"kubernetes.io/projected/9ffb5bd3-cb59-415a-b061-6084950220de-kube-api-access-zfbn4\") pod \"9ffb5bd3-cb59-415a-b061-6084950220de\" (UID: \"9ffb5bd3-cb59-415a-b061-6084950220de\") "
Sep 11 17:37:55.172157 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:55.172131    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31921d54-4a87-46f4-aed5-4954c6946bd3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "31921d54-4a87-46f4-aed5-4954c6946bd3" (UID: "31921d54-4a87-46f4-aed5-4954c6946bd3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:37:55.172301 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:55.172269    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ffb5bd3-cb59-415a-b061-6084950220de-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9ffb5bd3-cb59-415a-b061-6084950220de" (UID: "9ffb5bd3-cb59-415a-b061-6084950220de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:37:55.172301 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:55.172291    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31921d54-4a87-46f4-aed5-4954c6946bd3-plugins-config-volume\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:37:55.174466 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:55.174441    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31921d54-4a87-46f4-aed5-4954c6946bd3-kube-api-access-h46wb" (OuterVolumeSpecName: "kube-api-access-h46wb") pod "31921d54-4a87-46f4-aed5-4954c6946bd3" (UID: "31921d54-4a87-46f4-aed5-4954c6946bd3"). InnerVolumeSpecName "kube-api-access-h46wb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:37:55.174570 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:55.174511    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ffb5bd3-cb59-415a-b061-6084950220de-kube-api-access-zfbn4" (OuterVolumeSpecName: "kube-api-access-zfbn4") pod "9ffb5bd3-cb59-415a-b061-6084950220de" (UID: "9ffb5bd3-cb59-415a-b061-6084950220de"). InnerVolumeSpecName "kube-api-access-zfbn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:37:55.273383 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:55.273348    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h46wb\" (UniqueName: \"kubernetes.io/projected/31921d54-4a87-46f4-aed5-4954c6946bd3-kube-api-access-h46wb\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:37:55.273383 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:55.273381    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfbn4\" (UniqueName: \"kubernetes.io/projected/9ffb5bd3-cb59-415a-b061-6084950220de-kube-api-access-zfbn4\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:37:55.273599 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:55.273394    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ffb5bd3-cb59-415a-b061-6084950220de-plugins-config-volume\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:37:55.284639 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:55.284610    2562 generic.go:356] "Generic (PLEG): container finished" podID="9ffb5bd3-cb59-415a-b061-6084950220de" containerID="510aa455c9bd7d8c1199b82a8d8b55eeff2f396060775118f139f02e283e8839" exitCode=0
Sep 11 17:37:55.284795 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:55.284664    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ec3fed-567c4fd854-rt5x7"
Sep 11 17:37:55.284795 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:55.284702    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ec3fed-567c4fd854-rt5x7" event={"ID":"9ffb5bd3-cb59-415a-b061-6084950220de","Type":"ContainerDied","Data":"510aa455c9bd7d8c1199b82a8d8b55eeff2f396060775118f139f02e283e8839"}
Sep 11 17:37:55.284795 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:55.284759    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ec3fed-567c4fd854-rt5x7" event={"ID":"9ffb5bd3-cb59-415a-b061-6084950220de","Type":"ContainerDied","Data":"d96f49271c2dc5e55b28856f7177b1a5652305d3b869ed42b6e66a0fd9a3dd3a"}
Sep 11 17:37:55.284795 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:55.284788    2562 scope.go:117] "RemoveContainer" containerID="510aa455c9bd7d8c1199b82a8d8b55eeff2f396060775118f139f02e283e8839"
Sep 11 17:37:55.286179 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:55.286150    2562 generic.go:356] "Generic (PLEG): container finished" podID="31921d54-4a87-46f4-aed5-4954c6946bd3" containerID="bf17fbcf2911bc01d8dfe9ff3696d3b5d24dc3c5c2492ec4191a5f2b13c7fea4" exitCode=0
Sep 11 17:37:55.286296 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:55.286191    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-ec3fed-7865cdbb49-4dpfb"
Sep 11 17:37:55.286296 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:55.286221    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-ec3fed-7865cdbb49-4dpfb" event={"ID":"31921d54-4a87-46f4-aed5-4954c6946bd3","Type":"ContainerDied","Data":"bf17fbcf2911bc01d8dfe9ff3696d3b5d24dc3c5c2492ec4191a5f2b13c7fea4"}
Sep 11 17:37:55.286296 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:55.286286    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-ec3fed-7865cdbb49-4dpfb" event={"ID":"31921d54-4a87-46f4-aed5-4954c6946bd3","Type":"ContainerDied","Data":"ffc7aa8b105067a20c2cd46eccc152bc521e5102ffd4ae354f9c2be3a3ec4d93"}
Sep 11 17:37:55.295362 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:55.295346    2562 scope.go:117] "RemoveContainer" containerID="510aa455c9bd7d8c1199b82a8d8b55eeff2f396060775118f139f02e283e8839"
Sep 11 17:37:55.295654 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:37:55.295636    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"510aa455c9bd7d8c1199b82a8d8b55eeff2f396060775118f139f02e283e8839\": container with ID starting with 510aa455c9bd7d8c1199b82a8d8b55eeff2f396060775118f139f02e283e8839 not found: ID does not exist" containerID="510aa455c9bd7d8c1199b82a8d8b55eeff2f396060775118f139f02e283e8839"
Sep 11 17:37:55.295709 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:55.295663    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"510aa455c9bd7d8c1199b82a8d8b55eeff2f396060775118f139f02e283e8839"} err="failed to get container status \"510aa455c9bd7d8c1199b82a8d8b55eeff2f396060775118f139f02e283e8839\": rpc error: code = NotFound desc = could not find container \"510aa455c9bd7d8c1199b82a8d8b55eeff2f396060775118f139f02e283e8839\": container with ID starting with 510aa455c9bd7d8c1199b82a8d8b55eeff2f396060775118f139f02e283e8839 not found: ID does not exist"
Sep 11 17:37:55.295709 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:55.295696    2562 scope.go:117] "RemoveContainer" containerID="bf17fbcf2911bc01d8dfe9ff3696d3b5d24dc3c5c2492ec4191a5f2b13c7fea4"
Sep 11 17:37:55.303608 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:55.303594    2562 scope.go:117] "RemoveContainer" containerID="bf17fbcf2911bc01d8dfe9ff3696d3b5d24dc3c5c2492ec4191a5f2b13c7fea4"
Sep 11 17:37:55.303874 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:37:55.303856    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf17fbcf2911bc01d8dfe9ff3696d3b5d24dc3c5c2492ec4191a5f2b13c7fea4\": container with ID starting with bf17fbcf2911bc01d8dfe9ff3696d3b5d24dc3c5c2492ec4191a5f2b13c7fea4 not found: ID does not exist" containerID="bf17fbcf2911bc01d8dfe9ff3696d3b5d24dc3c5c2492ec4191a5f2b13c7fea4"
Sep 11 17:37:55.303937 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:55.303879    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf17fbcf2911bc01d8dfe9ff3696d3b5d24dc3c5c2492ec4191a5f2b13c7fea4"} err="failed to get container status \"bf17fbcf2911bc01d8dfe9ff3696d3b5d24dc3c5c2492ec4191a5f2b13c7fea4\": rpc error: code = NotFound desc = could not find container \"bf17fbcf2911bc01d8dfe9ff3696d3b5d24dc3c5c2492ec4191a5f2b13c7fea4\": container with ID starting with bf17fbcf2911bc01d8dfe9ff3696d3b5d24dc3c5c2492ec4191a5f2b13c7fea4 not found: ID does not exist"
Sep 11 17:37:55.334137 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:55.334113    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ec3fed-567c4fd854-rt5x7"]
Sep 11 17:37:55.343987 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:55.343955    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ec3fed-567c4fd854-rt5x7"]
Sep 11 17:37:55.375008 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:55.374967    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ec3fed-7865cdbb49-4dpfb"]
Sep 11 17:37:55.379391 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:55.379371    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ec3fed-7865cdbb49-4dpfb"]
Sep 11 17:37:56.125979 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:56.125946    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31921d54-4a87-46f4-aed5-4954c6946bd3" path="/var/lib/kubelet/pods/31921d54-4a87-46f4-aed5-4954c6946bd3/volumes"
Sep 11 17:37:56.126365 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:56.126352    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ffb5bd3-cb59-415a-b061-6084950220de" path="/var/lib/kubelet/pods/9ffb5bd3-cb59-415a-b061-6084950220de/volumes"
Sep 11 17:37:57.128591 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.128559    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"]
Sep 11 17:37:57.128957 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.128936    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31921d54-4a87-46f4-aed5-4954c6946bd3" containerName="payload-processing"
Sep 11 17:37:57.128957 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.128948    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="31921d54-4a87-46f4-aed5-4954c6946bd3" containerName="payload-processing"
Sep 11 17:37:57.129047 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.128973    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ffb5bd3-cb59-415a-b061-6084950220de" containerName="payload-pre-processing"
Sep 11 17:37:57.129047 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.128979    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ffb5bd3-cb59-415a-b061-6084950220de" containerName="payload-pre-processing"
Sep 11 17:37:57.129119 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.129061    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ffb5bd3-cb59-415a-b061-6084950220de" containerName="payload-pre-processing"
Sep 11 17:37:57.129119 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.129071    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="31921d54-4a87-46f4-aed5-4954c6946bd3" containerName="payload-processing"
Sep 11 17:37:57.133622 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.133603    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"
Sep 11 17:37:57.138761 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.138745    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6709f1a7-openshift-default-dockercfg-x772j\""
Sep 11 17:37:57.170211 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.170181    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"]
Sep 11 17:37:57.189387 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.189353    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5eab92b-01a6-4f8e-a118-04d2db7fb047-workload-socket\") pod \"e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql\" (UID: \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\") " pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"
Sep 11 17:37:57.189542 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.189411    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5eab92b-01a6-4f8e-a118-04d2db7fb047-istiod-ca-cert\") pod \"e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql\" (UID: \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\") " pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"
Sep 11 17:37:57.189542 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.189489    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5eab92b-01a6-4f8e-a118-04d2db7fb047-workload-certs\") pod \"e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql\" (UID: \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\") " pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"
Sep 11 17:37:57.189542 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.189531    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5eab92b-01a6-4f8e-a118-04d2db7fb047-credential-socket\") pod \"e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql\" (UID: \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\") " pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"
Sep 11 17:37:57.189670 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.189554    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5eab92b-01a6-4f8e-a118-04d2db7fb047-istio-podinfo\") pod \"e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql\" (UID: \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\") " pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"
Sep 11 17:37:57.189670 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.189576    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5eab92b-01a6-4f8e-a118-04d2db7fb047-istio-data\") pod \"e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql\" (UID: \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\") " pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"
Sep 11 17:37:57.189670 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.189641    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd75p\" (UniqueName: \"kubernetes.io/projected/e5eab92b-01a6-4f8e-a118-04d2db7fb047-kube-api-access-gd75p\") pod \"e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql\" (UID: \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\") " pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"
Sep 11 17:37:57.189786 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.189680    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5eab92b-01a6-4f8e-a118-04d2db7fb047-istio-envoy\") pod \"e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql\" (UID: \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\") " pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"
Sep 11 17:37:57.189786 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.189768    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5eab92b-01a6-4f8e-a118-04d2db7fb047-istio-token\") pod \"e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql\" (UID: \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\") " pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"
Sep 11 17:37:57.290718 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.290689    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5eab92b-01a6-4f8e-a118-04d2db7fb047-istiod-ca-cert\") pod \"e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql\" (UID: \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\") " pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"
Sep 11 17:37:57.290899 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.290729    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5eab92b-01a6-4f8e-a118-04d2db7fb047-workload-certs\") pod \"e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql\" (UID: \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\") " pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"
Sep 11 17:37:57.290899 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.290756    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5eab92b-01a6-4f8e-a118-04d2db7fb047-credential-socket\") pod \"e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql\" (UID: \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\") " pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"
Sep 11 17:37:57.290899 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.290786    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5eab92b-01a6-4f8e-a118-04d2db7fb047-istio-podinfo\") pod \"e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql\" (UID: \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\") " pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"
Sep 11 17:37:57.290899 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.290817    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5eab92b-01a6-4f8e-a118-04d2db7fb047-istio-data\") pod \"e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql\" (UID: \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\") " pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"
Sep 11 17:37:57.290899 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.290851    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd75p\" (UniqueName: \"kubernetes.io/projected/e5eab92b-01a6-4f8e-a118-04d2db7fb047-kube-api-access-gd75p\") pod \"e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql\" (UID: \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\") " pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"
Sep 11 17:37:57.291209 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.291169    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5eab92b-01a6-4f8e-a118-04d2db7fb047-istio-envoy\") pod \"e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql\" (UID: \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\") " pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"
Sep 11 17:37:57.291266 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.291206    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5eab92b-01a6-4f8e-a118-04d2db7fb047-workload-certs\") pod \"e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql\" (UID: \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\") " pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"
Sep 11 17:37:57.291353 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.291329    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5eab92b-01a6-4f8e-a118-04d2db7fb047-istio-token\") pod \"e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql\" (UID: \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\") " pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"
Sep 11 17:37:57.291483 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.291366    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5eab92b-01a6-4f8e-a118-04d2db7fb047-istio-data\") pod \"e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql\" (UID: \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\") " pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"
Sep 11 17:37:57.291548 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.291523    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5eab92b-01a6-4f8e-a118-04d2db7fb047-istiod-ca-cert\") pod \"e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql\" (UID: \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\") " pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"
Sep 11 17:37:57.291636 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.291612    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5eab92b-01a6-4f8e-a118-04d2db7fb047-credential-socket\") pod \"e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql\" (UID: \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\") " pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"
Sep 11 17:37:57.291704 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.291641    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5eab92b-01a6-4f8e-a118-04d2db7fb047-workload-socket\") pod \"e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql\" (UID: \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\") " pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"
Sep 11 17:37:57.292033 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.291956    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5eab92b-01a6-4f8e-a118-04d2db7fb047-workload-socket\") pod \"e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql\" (UID: \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\") " pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"
Sep 11 17:37:57.293375 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.293354    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5eab92b-01a6-4f8e-a118-04d2db7fb047-istio-envoy\") pod \"e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql\" (UID: \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\") " pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"
Sep 11 17:37:57.293813 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.293791    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5eab92b-01a6-4f8e-a118-04d2db7fb047-istio-podinfo\") pod \"e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql\" (UID: \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\") " pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"
Sep 11 17:37:57.316266 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.316240    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd75p\" (UniqueName: \"kubernetes.io/projected/e5eab92b-01a6-4f8e-a118-04d2db7fb047-kube-api-access-gd75p\") pod \"e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql\" (UID: \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\") " pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"
Sep 11 17:37:57.317918 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.317901    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5eab92b-01a6-4f8e-a118-04d2db7fb047-istio-token\") pod \"e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql\" (UID: \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\") " pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"
Sep 11 17:37:57.444078 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.443981    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"
Sep 11 17:37:57.733224 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.733194    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"]
Sep 11 17:37:57.735662 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:37:57.735628    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5eab92b_01a6_4f8e_a118_04d2db7fb047.slice/crio-32f439f4f107e0c15c1fa595446bf4ff7f8fb6e636a261c6dd810e9a5e186153 WatchSource:0}: Error finding container 32f439f4f107e0c15c1fa595446bf4ff7f8fb6e636a261c6dd810e9a5e186153: Status 404 returned error can't find the container with id 32f439f4f107e0c15c1fa595446bf4ff7f8fb6e636a261c6dd810e9a5e186153
Sep 11 17:37:57.738065 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.738034    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:37:57.738145 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.738098    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:37:57.738145 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:57.738126    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:37:58.306538 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:58.306505    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql" event={"ID":"e5eab92b-01a6-4f8e-a118-04d2db7fb047","Type":"ContainerStarted","Data":"81b6fa74b628ddea2f75a92623ce2bb97c698b6ae130ddbfed68ffdd92036003"}
Sep 11 17:37:58.306943 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:58.306557    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql" event={"ID":"e5eab92b-01a6-4f8e-a118-04d2db7fb047","Type":"ContainerStarted","Data":"32f439f4f107e0c15c1fa595446bf4ff7f8fb6e636a261c6dd810e9a5e186153"}
Sep 11 17:37:58.333793 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:58.333734    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql" podStartSLOduration=1.333708591 podStartE2EDuration="1.333708591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:37:58.329936263 +0000 UTC m=+1158.723853718" watchObservedRunningTime="2026-09-11 17:37:58.333708591 +0000 UTC m=+1158.727626045"
Sep 11 17:37:58.444730 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:58.444638    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"
Sep 11 17:37:58.455966 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:58.455934    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"
Sep 11 17:37:59.312159 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:59.312125    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"
Sep 11 17:37:59.312967 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:37:59.312951    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"
Sep 11 17:38:03.726930 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:03.726849    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6709f1a7-55c86c99d-gfvl6"]
Sep 11 17:38:03.730604 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:03.730587    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6709f1a7-55c86c99d-gfvl6"
Sep 11 17:38:03.733256 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:03.733233    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6709f1a7-dockercfg-sc2gj\""
Sep 11 17:38:03.733256 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:03.733248    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6709f1a7\""
Sep 11 17:38:03.746410 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:03.746384    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6709f1a7-55c86c99d-gfvl6"]
Sep 11 17:38:03.855575 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:03.855534    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdqfs\" (UniqueName: \"kubernetes.io/projected/67502d7f-4095-4544-8561-231e3b23ceac-kube-api-access-fdqfs\") pod \"payload-pre-processing-e2e-mt-6709f1a7-55c86c99d-gfvl6\" (UID: \"67502d7f-4095-4544-8561-231e3b23ceac\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6709f1a7-55c86c99d-gfvl6"
Sep 11 17:38:03.855575 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:03.855576    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67502d7f-4095-4544-8561-231e3b23ceac-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6709f1a7-55c86c99d-gfvl6\" (UID: \"67502d7f-4095-4544-8561-231e3b23ceac\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6709f1a7-55c86c99d-gfvl6"
Sep 11 17:38:03.956799 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:03.956757    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdqfs\" (UniqueName: \"kubernetes.io/projected/67502d7f-4095-4544-8561-231e3b23ceac-kube-api-access-fdqfs\") pod \"payload-pre-processing-e2e-mt-6709f1a7-55c86c99d-gfvl6\" (UID: \"67502d7f-4095-4544-8561-231e3b23ceac\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6709f1a7-55c86c99d-gfvl6"
Sep 11 17:38:03.956984 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:03.956816    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67502d7f-4095-4544-8561-231e3b23ceac-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6709f1a7-55c86c99d-gfvl6\" (UID: \"67502d7f-4095-4544-8561-231e3b23ceac\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6709f1a7-55c86c99d-gfvl6"
Sep 11 17:38:03.957690 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:03.957665    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67502d7f-4095-4544-8561-231e3b23ceac-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6709f1a7-55c86c99d-gfvl6\" (UID: \"67502d7f-4095-4544-8561-231e3b23ceac\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6709f1a7-55c86c99d-gfvl6"
Sep 11 17:38:03.967478 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:03.967447    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdqfs\" (UniqueName: \"kubernetes.io/projected/67502d7f-4095-4544-8561-231e3b23ceac-kube-api-access-fdqfs\") pod \"payload-pre-processing-e2e-mt-6709f1a7-55c86c99d-gfvl6\" (UID: \"67502d7f-4095-4544-8561-231e3b23ceac\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6709f1a7-55c86c99d-gfvl6"
Sep 11 17:38:04.042076 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:04.041963    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6709f1a7-55c86c99d-gfvl6"
Sep 11 17:38:04.319661 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:04.319611    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6709f1a7-55c86c99d-gfvl6"]
Sep 11 17:38:04.322008 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:38:04.321968    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67502d7f_4095_4544_8561_231e3b23ceac.slice/crio-4c0d1178762cd2face10932de6acf3ceacf54faa9aa2cd7f88574587b16d5f6d WatchSource:0}: Error finding container 4c0d1178762cd2face10932de6acf3ceacf54faa9aa2cd7f88574587b16d5f6d: Status 404 returned error can't find the container with id 4c0d1178762cd2face10932de6acf3ceacf54faa9aa2cd7f88574587b16d5f6d
Sep 11 17:38:04.343119 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:04.343095    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6709f1a7-55c86c99d-gfvl6" event={"ID":"67502d7f-4095-4544-8561-231e3b23ceac","Type":"ContainerStarted","Data":"4c0d1178762cd2face10932de6acf3ceacf54faa9aa2cd7f88574587b16d5f6d"}
Sep 11 17:38:05.349374 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:05.349337    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6709f1a7-55c86c99d-gfvl6" event={"ID":"67502d7f-4095-4544-8561-231e3b23ceac","Type":"ContainerStarted","Data":"bf2cffd0db5c4249587cb51d46b331ff1155948baaaf57f8467147812d71cae2"}
Sep 11 17:38:05.349825 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:05.349403    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6709f1a7-55c86c99d-gfvl6"
Sep 11 17:38:05.377269 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:05.377224    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-6709f1a7-55c86c99d-gfvl6" podStartSLOduration=2.377207363 podStartE2EDuration="2.377207363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:38:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:38:05.376084038 +0000 UTC m=+1165.770001489" watchObservedRunningTime="2026-09-11 17:38:05.377207363 +0000 UTC m=+1165.771124817"
Sep 11 17:38:10.282282 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:10.282245    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"]
Sep 11 17:38:10.285864 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:10.285842    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"
Sep 11 17:38:10.289768 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:10.289733    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-ac1a16-openshift-default-dockercfg-n98vl\""
Sep 11 17:38:10.375872 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:10.375841    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"]
Sep 11 17:38:10.428257 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:10.428218    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-istio-token\") pod \"e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh\" (UID: \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\") " pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"
Sep 11 17:38:10.428465 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:10.428268    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-istio-podinfo\") pod \"e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh\" (UID: \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\") " pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"
Sep 11 17:38:10.428465 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:10.428331    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wbbg\" (UniqueName: \"kubernetes.io/projected/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-kube-api-access-6wbbg\") pod \"e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh\" (UID: \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\") " pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"
Sep 11 17:38:10.428465 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:10.428401    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-workload-certs\") pod \"e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh\" (UID: \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\") " pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"
Sep 11 17:38:10.428465 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:10.428428    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-istio-data\") pod \"e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh\" (UID: \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\") " pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"
Sep 11 17:38:10.428465 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:10.428457    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-istiod-ca-cert\") pod \"e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh\" (UID: \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\") " pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"
Sep 11 17:38:10.428677 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:10.428553    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-workload-socket\") pod \"e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh\" (UID: \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\") " pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"
Sep 11 17:38:10.428677 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:10.428585    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-istio-envoy\") pod \"e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh\" (UID: \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\") " pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"
Sep 11 17:38:10.428677 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:10.428612    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-credential-socket\") pod \"e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh\" (UID: \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\") " pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"
Sep 11 17:38:10.529943 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:10.529908    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-istio-token\") pod \"e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh\" (UID: \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\") " pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"
Sep 11 17:38:10.530177 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:10.529952    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-istio-podinfo\") pod \"e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh\" (UID: \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\") " pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"
Sep 11 17:38:10.530177 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:10.529976    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wbbg\" (UniqueName: \"kubernetes.io/projected/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-kube-api-access-6wbbg\") pod \"e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh\" (UID: \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\") " pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"
Sep 11 17:38:10.530177 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:10.530027    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-workload-certs\") pod \"e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh\" (UID: \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\") " pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"
Sep 11 17:38:10.530177 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:10.530050    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-istio-data\") pod \"e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh\" (UID: \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\") " pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"
Sep 11 17:38:10.530177 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:10.530068    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-istiod-ca-cert\") pod \"e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh\" (UID: \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\") " pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"
Sep 11 17:38:10.530177 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:10.530124    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-workload-socket\") pod \"e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh\" (UID: \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\") " pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"
Sep 11 17:38:10.530177 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:10.530159    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-istio-envoy\") pod \"e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh\" (UID: \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\") " pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"
Sep 11 17:38:10.530559 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:10.530193    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-credential-socket\") pod \"e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh\" (UID: \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\") " pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"
Sep 11 17:38:10.530617 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:10.530549    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-workload-socket\") pod \"e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh\" (UID: \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\") " pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"
Sep 11 17:38:10.530617 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:10.530564    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-workload-certs\") pod \"e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh\" (UID: \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\") " pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"
Sep 11 17:38:10.530721 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:10.530655    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-credential-socket\") pod \"e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh\" (UID: \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\") " pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"
Sep 11 17:38:10.530721 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:10.530689    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-istio-data\") pod \"e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh\" (UID: \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\") " pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"
Sep 11 17:38:10.530902 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:10.530884    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-istiod-ca-cert\") pod \"e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh\" (UID: \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\") " pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"
Sep 11 17:38:10.532603 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:10.532554    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-istio-podinfo\") pod \"e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh\" (UID: \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\") " pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"
Sep 11 17:38:10.532670 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:10.532602    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-istio-envoy\") pod \"e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh\" (UID: \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\") " pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"
Sep 11 17:38:10.540299 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:10.540279    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wbbg\" (UniqueName: \"kubernetes.io/projected/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-kube-api-access-6wbbg\") pod \"e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh\" (UID: \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\") " pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"
Sep 11 17:38:10.543292 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:10.543269    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-istio-token\") pod \"e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh\" (UID: \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\") " pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"
Sep 11 17:38:10.597048 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:10.596986    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"
Sep 11 17:38:10.900066 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:10.900041    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"]
Sep 11 17:38:10.901061 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:38:10.901033    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc6aca4_5bd3_4520_b0d7_ba46df2890a4.slice/crio-965247c08ee5bf38fc1e32c3a4482e644baf1c7a5f4c48fe454de1cbea05199c WatchSource:0}: Error finding container 965247c08ee5bf38fc1e32c3a4482e644baf1c7a5f4c48fe454de1cbea05199c: Status 404 returned error can't find the container with id 965247c08ee5bf38fc1e32c3a4482e644baf1c7a5f4c48fe454de1cbea05199c
Sep 11 17:38:10.903284 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:10.903251    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:38:10.903376 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:10.903314    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:38:10.903376 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:10.903343    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:38:11.381465 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:11.381433    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh" event={"ID":"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4","Type":"ContainerStarted","Data":"e6514fd8444a43a9fc70bf58eef853d058ae76b01ef3fb2f9b17798f4ede1df7"}
Sep 11 17:38:11.381879 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:11.381486    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh" event={"ID":"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4","Type":"ContainerStarted","Data":"965247c08ee5bf38fc1e32c3a4482e644baf1c7a5f4c48fe454de1cbea05199c"}
Sep 11 17:38:11.407661 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:11.407607    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh" podStartSLOduration=1.40758797 podStartE2EDuration="1.40758797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:38:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:38:11.404733609 +0000 UTC m=+1171.798651063" watchObservedRunningTime="2026-09-11 17:38:11.40758797 +0000 UTC m=+1171.801505427"
Sep 11 17:38:11.597534 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:11.597506    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"
Sep 11 17:38:11.602652 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:11.602627    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"
Sep 11 17:38:12.387032 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:12.386977    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"
Sep 11 17:38:12.387945 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:12.387926    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"
Sep 11 17:38:16.357181 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:16.357147    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6709f1a7-55c86c99d-gfvl6"
Sep 11 17:38:25.465547 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:25.465516    2562 generic.go:356] "Generic (PLEG): container finished" podID="e5eab92b-01a6-4f8e-a118-04d2db7fb047" containerID="81b6fa74b628ddea2f75a92623ce2bb97c698b6ae130ddbfed68ffdd92036003" exitCode=0
Sep 11 17:38:25.466017 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:25.465592    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql" event={"ID":"e5eab92b-01a6-4f8e-a118-04d2db7fb047","Type":"ContainerDied","Data":"81b6fa74b628ddea2f75a92623ce2bb97c698b6ae130ddbfed68ffdd92036003"}
Sep 11 17:38:25.466153 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:25.466136    2562 scope.go:117] "RemoveContainer" containerID="81b6fa74b628ddea2f75a92623ce2bb97c698b6ae130ddbfed68ffdd92036003"
Sep 11 17:38:25.467432 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:25.467400    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:38:25.467548 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:25.467502    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:38:25.467612 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:25.467555    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:38:26.473963 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:26.473929    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql" event={"ID":"e5eab92b-01a6-4f8e-a118-04d2db7fb047","Type":"ContainerStarted","Data":"1a1d1a0f8ef94520bda9310bd172742ffccf798b3937086da000db2ff6bb63c1"}
Sep 11 17:38:27.452021 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:27.448391    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"
Sep 11 17:38:27.460147 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:27.460107    2562 patch_prober.go:28] interesting pod/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 11 17:38:27.460417 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:27.460383    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql" podUID="e5eab92b-01a6-4f8e-a118-04d2db7fb047" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 11 17:38:28.445137 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:28.445095    2562 patch_prober.go:28] interesting pod/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 11 17:38:28.445624 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:28.445158    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql" podUID="e5eab92b-01a6-4f8e-a118-04d2db7fb047" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 11 17:38:29.312102 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:29.312045    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"
Sep 11 17:38:29.444957 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:29.444921    2562 patch_prober.go:28] interesting pod/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 11 17:38:29.445173 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:29.445009    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql" podUID="e5eab92b-01a6-4f8e-a118-04d2db7fb047" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 11 17:38:30.445044 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:30.444988    2562 patch_prober.go:28] interesting pod/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 11 17:38:30.445258 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:30.445075    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql" podUID="e5eab92b-01a6-4f8e-a118-04d2db7fb047" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 11 17:38:31.445012 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:31.444952    2562 patch_prober.go:28] interesting pod/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 11 17:38:31.445232 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:31.445047    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql" podUID="e5eab92b-01a6-4f8e-a118-04d2db7fb047" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 11 17:38:32.449296 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:32.449262    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"
Sep 11 17:38:32.450280 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:32.450263    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"
Sep 11 17:38:51.694436 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:51.694402    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-ac1a16/noten-model-ac1a16-kserve-78dc66fdcf-bx7m8"]
Sep 11 17:38:51.697738 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:51.697717    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ac1a16/noten-model-ac1a16-kserve-78dc66fdcf-bx7m8"
Sep 11 17:38:51.700139 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:51.700117    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-ac1a16\"/\"kube-root-ca.crt\""
Sep 11 17:38:51.700712 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:51.700693    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-ac1a16\"/\"default-dockercfg-p42fl\""
Sep 11 17:38:51.700814 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:51.700744    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-ac1a16\"/\"noten-model-ac1a16-kserve-self-signed-certs\""
Sep 11 17:38:51.700814 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:51.700756    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-ac1a16\"/\"openshift-service-ca.crt\""
Sep 11 17:38:51.709240 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:51.709217    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-ac1a16/noten-model-ac1a16-kserve-78dc66fdcf-bx7m8"]
Sep 11 17:38:51.849839 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:51.849800    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc3d82f7-197f-47e6-8cd0-d229c3fd18fd-tls-certs\") pod \"noten-model-ac1a16-kserve-78dc66fdcf-bx7m8\" (UID: \"bc3d82f7-197f-47e6-8cd0-d229c3fd18fd\") " pod="ai-tenant-e2e-noten-ac1a16/noten-model-ac1a16-kserve-78dc66fdcf-bx7m8"
Sep 11 17:38:51.850066 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:51.849869    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc3d82f7-197f-47e6-8cd0-d229c3fd18fd-model-cache\") pod \"noten-model-ac1a16-kserve-78dc66fdcf-bx7m8\" (UID: \"bc3d82f7-197f-47e6-8cd0-d229c3fd18fd\") " pod="ai-tenant-e2e-noten-ac1a16/noten-model-ac1a16-kserve-78dc66fdcf-bx7m8"
Sep 11 17:38:51.850066 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:51.849902    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z56h\" (UniqueName: \"kubernetes.io/projected/bc3d82f7-197f-47e6-8cd0-d229c3fd18fd-kube-api-access-7z56h\") pod \"noten-model-ac1a16-kserve-78dc66fdcf-bx7m8\" (UID: \"bc3d82f7-197f-47e6-8cd0-d229c3fd18fd\") " pod="ai-tenant-e2e-noten-ac1a16/noten-model-ac1a16-kserve-78dc66fdcf-bx7m8"
Sep 11 17:38:51.850066 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:51.849926    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc3d82f7-197f-47e6-8cd0-d229c3fd18fd-dshm\") pod \"noten-model-ac1a16-kserve-78dc66fdcf-bx7m8\" (UID: \"bc3d82f7-197f-47e6-8cd0-d229c3fd18fd\") " pod="ai-tenant-e2e-noten-ac1a16/noten-model-ac1a16-kserve-78dc66fdcf-bx7m8"
Sep 11 17:38:51.850066 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:51.850052    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc3d82f7-197f-47e6-8cd0-d229c3fd18fd-tmp-dir\") pod \"noten-model-ac1a16-kserve-78dc66fdcf-bx7m8\" (UID: \"bc3d82f7-197f-47e6-8cd0-d229c3fd18fd\") " pod="ai-tenant-e2e-noten-ac1a16/noten-model-ac1a16-kserve-78dc66fdcf-bx7m8"
Sep 11 17:38:51.850299 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:51.850093    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc3d82f7-197f-47e6-8cd0-d229c3fd18fd-home\") pod \"noten-model-ac1a16-kserve-78dc66fdcf-bx7m8\" (UID: \"bc3d82f7-197f-47e6-8cd0-d229c3fd18fd\") " pod="ai-tenant-e2e-noten-ac1a16/noten-model-ac1a16-kserve-78dc66fdcf-bx7m8"
Sep 11 17:38:51.951355 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:51.951259    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z56h\" (UniqueName: \"kubernetes.io/projected/bc3d82f7-197f-47e6-8cd0-d229c3fd18fd-kube-api-access-7z56h\") pod \"noten-model-ac1a16-kserve-78dc66fdcf-bx7m8\" (UID: \"bc3d82f7-197f-47e6-8cd0-d229c3fd18fd\") " pod="ai-tenant-e2e-noten-ac1a16/noten-model-ac1a16-kserve-78dc66fdcf-bx7m8"
Sep 11 17:38:51.951355 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:51.951300    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc3d82f7-197f-47e6-8cd0-d229c3fd18fd-dshm\") pod \"noten-model-ac1a16-kserve-78dc66fdcf-bx7m8\" (UID: \"bc3d82f7-197f-47e6-8cd0-d229c3fd18fd\") " pod="ai-tenant-e2e-noten-ac1a16/noten-model-ac1a16-kserve-78dc66fdcf-bx7m8"
Sep 11 17:38:51.951355 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:51.951350    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc3d82f7-197f-47e6-8cd0-d229c3fd18fd-tmp-dir\") pod \"noten-model-ac1a16-kserve-78dc66fdcf-bx7m8\" (UID: \"bc3d82f7-197f-47e6-8cd0-d229c3fd18fd\") " pod="ai-tenant-e2e-noten-ac1a16/noten-model-ac1a16-kserve-78dc66fdcf-bx7m8"
Sep 11 17:38:51.951643 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:51.951390    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc3d82f7-197f-47e6-8cd0-d229c3fd18fd-home\") pod \"noten-model-ac1a16-kserve-78dc66fdcf-bx7m8\" (UID: \"bc3d82f7-197f-47e6-8cd0-d229c3fd18fd\") " pod="ai-tenant-e2e-noten-ac1a16/noten-model-ac1a16-kserve-78dc66fdcf-bx7m8"
Sep 11 17:38:51.951643 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:51.951496    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc3d82f7-197f-47e6-8cd0-d229c3fd18fd-tls-certs\") pod \"noten-model-ac1a16-kserve-78dc66fdcf-bx7m8\" (UID: \"bc3d82f7-197f-47e6-8cd0-d229c3fd18fd\") " pod="ai-tenant-e2e-noten-ac1a16/noten-model-ac1a16-kserve-78dc66fdcf-bx7m8"
Sep 11 17:38:51.951643 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:51.951550    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc3d82f7-197f-47e6-8cd0-d229c3fd18fd-model-cache\") pod \"noten-model-ac1a16-kserve-78dc66fdcf-bx7m8\" (UID: \"bc3d82f7-197f-47e6-8cd0-d229c3fd18fd\") " pod="ai-tenant-e2e-noten-ac1a16/noten-model-ac1a16-kserve-78dc66fdcf-bx7m8"
Sep 11 17:38:51.951871 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:51.951832    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc3d82f7-197f-47e6-8cd0-d229c3fd18fd-tmp-dir\") pod \"noten-model-ac1a16-kserve-78dc66fdcf-bx7m8\" (UID: \"bc3d82f7-197f-47e6-8cd0-d229c3fd18fd\") " pod="ai-tenant-e2e-noten-ac1a16/noten-model-ac1a16-kserve-78dc66fdcf-bx7m8"
Sep 11 17:38:51.952007 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:51.951846    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc3d82f7-197f-47e6-8cd0-d229c3fd18fd-home\") pod \"noten-model-ac1a16-kserve-78dc66fdcf-bx7m8\" (UID: \"bc3d82f7-197f-47e6-8cd0-d229c3fd18fd\") " pod="ai-tenant-e2e-noten-ac1a16/noten-model-ac1a16-kserve-78dc66fdcf-bx7m8"
Sep 11 17:38:51.952007 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:51.951928    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc3d82f7-197f-47e6-8cd0-d229c3fd18fd-model-cache\") pod \"noten-model-ac1a16-kserve-78dc66fdcf-bx7m8\" (UID: \"bc3d82f7-197f-47e6-8cd0-d229c3fd18fd\") " pod="ai-tenant-e2e-noten-ac1a16/noten-model-ac1a16-kserve-78dc66fdcf-bx7m8"
Sep 11 17:38:51.953609 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:51.953582    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc3d82f7-197f-47e6-8cd0-d229c3fd18fd-dshm\") pod \"noten-model-ac1a16-kserve-78dc66fdcf-bx7m8\" (UID: \"bc3d82f7-197f-47e6-8cd0-d229c3fd18fd\") " pod="ai-tenant-e2e-noten-ac1a16/noten-model-ac1a16-kserve-78dc66fdcf-bx7m8"
Sep 11 17:38:51.953869 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:51.953848    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc3d82f7-197f-47e6-8cd0-d229c3fd18fd-tls-certs\") pod \"noten-model-ac1a16-kserve-78dc66fdcf-bx7m8\" (UID: \"bc3d82f7-197f-47e6-8cd0-d229c3fd18fd\") " pod="ai-tenant-e2e-noten-ac1a16/noten-model-ac1a16-kserve-78dc66fdcf-bx7m8"
Sep 11 17:38:51.958818 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:51.958796    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z56h\" (UniqueName: \"kubernetes.io/projected/bc3d82f7-197f-47e6-8cd0-d229c3fd18fd-kube-api-access-7z56h\") pod \"noten-model-ac1a16-kserve-78dc66fdcf-bx7m8\" (UID: \"bc3d82f7-197f-47e6-8cd0-d229c3fd18fd\") " pod="ai-tenant-e2e-noten-ac1a16/noten-model-ac1a16-kserve-78dc66fdcf-bx7m8"
Sep 11 17:38:52.006692 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:52.006636    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ac1a16/noten-model-ac1a16-kserve-78dc66fdcf-bx7m8"
Sep 11 17:38:52.277626 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:52.277595    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-ac1a16/noten-model-ac1a16-kserve-78dc66fdcf-bx7m8"]
Sep 11 17:38:52.279794 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:38:52.279761    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc3d82f7_197f_47e6_8cd0_d229c3fd18fd.slice/crio-3095d9997558d05be67e540826e92193354d1b25b802e4e18526381087751b27 WatchSource:0}: Error finding container 3095d9997558d05be67e540826e92193354d1b25b802e4e18526381087751b27: Status 404 returned error can't find the container with id 3095d9997558d05be67e540826e92193354d1b25b802e4e18526381087751b27
Sep 11 17:38:52.622283 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:52.622248    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ac1a16/noten-model-ac1a16-kserve-78dc66fdcf-bx7m8" event={"ID":"bc3d82f7-197f-47e6-8cd0-d229c3fd18fd","Type":"ContainerStarted","Data":"489ab4ae7f440a4e7cf2d1fa78bdb19a89e77f5aa8dea9a5c8428737626aa2e6"}
Sep 11 17:38:52.622457 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:52.622291    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ac1a16/noten-model-ac1a16-kserve-78dc66fdcf-bx7m8" event={"ID":"bc3d82f7-197f-47e6-8cd0-d229c3fd18fd","Type":"ContainerStarted","Data":"3095d9997558d05be67e540826e92193354d1b25b802e4e18526381087751b27"}
Sep 11 17:38:52.642265 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:38:52.642217    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-ac1a16/noten-model-ac1a16-kserve-78dc66fdcf-bx7m8" podStartSLOduration=1.642201485 podStartE2EDuration="1.642201485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:38:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:38:52.640133951 +0000 UTC m=+1213.034051405" watchObservedRunningTime="2026-09-11 17:38:52.642201485 +0000 UTC m=+1213.036118940"
Sep 11 17:39:02.007644 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:02.007585    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-ac1a16/noten-model-ac1a16-kserve-78dc66fdcf-bx7m8"
Sep 11 17:39:02.007644 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:02.007660    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-ac1a16/noten-model-ac1a16-kserve-78dc66fdcf-bx7m8"
Sep 11 17:39:02.020093 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:02.020058    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-ac1a16/noten-model-ac1a16-kserve-78dc66fdcf-bx7m8"
Sep 11 17:39:02.684634 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:02.684605    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-ac1a16/noten-model-ac1a16-kserve-78dc66fdcf-bx7m8"
Sep 11 17:39:09.082888 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:09.082853    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cbbb09/test-model-cbbb09-kserve-d4dcf7cb5-mrgdn"]
Sep 11 17:39:18.357101 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:18.357067    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w"]
Sep 11 17:39:18.358843 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:18.358775    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w" podUID="5f249ed3-acb4-4e5d-b189-de21af9f1a25" containerName="maas-api" containerID="cri-o://6988d1aeb216e89cc487af686998a28876712a9c1308f80e9b9e1d813bc3c5c0" gracePeriod=30
Sep 11 17:39:18.432323 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:18.432289    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-08c6b2-6df7cb9cd-q9mx5"]
Sep 11 17:39:18.432541 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:18.432515    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08c6b2-6df7cb9cd-q9mx5" podUID="8d74b25a-7fd4-489c-943b-8fd444b32f38" containerName="payload-pre-processing" containerID="cri-o://a7c36504d070c824123abc943d8e516098e8cef56f089f8e2ebd22fb4eb58a21" gracePeriod=30
Sep 11 17:39:18.721380 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:18.721346    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w"
Sep 11 17:39:18.745464 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:18.745434    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08c6b2-6df7cb9cd-q9mx5"
Sep 11 17:39:18.752348 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:18.752318    2562 generic.go:356] "Generic (PLEG): container finished" podID="5f249ed3-acb4-4e5d-b189-de21af9f1a25" containerID="6988d1aeb216e89cc487af686998a28876712a9c1308f80e9b9e1d813bc3c5c0" exitCode=0
Sep 11 17:39:18.752504 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:18.752367    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w"
Sep 11 17:39:18.752504 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:18.752404    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w" event={"ID":"5f249ed3-acb4-4e5d-b189-de21af9f1a25","Type":"ContainerDied","Data":"6988d1aeb216e89cc487af686998a28876712a9c1308f80e9b9e1d813bc3c5c0"}
Sep 11 17:39:18.752504 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:18.752459    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w" event={"ID":"5f249ed3-acb4-4e5d-b189-de21af9f1a25","Type":"ContainerDied","Data":"bb230af9813967f766c16e8ef5f4e10edd7256c34c3083cbab9267d154011e1b"}
Sep 11 17:39:18.752504 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:18.752492    2562 scope.go:117] "RemoveContainer" containerID="6988d1aeb216e89cc487af686998a28876712a9c1308f80e9b9e1d813bc3c5c0"
Sep 11 17:39:18.753942 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:18.753917    2562 generic.go:356] "Generic (PLEG): container finished" podID="8d74b25a-7fd4-489c-943b-8fd444b32f38" containerID="a7c36504d070c824123abc943d8e516098e8cef56f089f8e2ebd22fb4eb58a21" exitCode=0
Sep 11 17:39:18.754083 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:18.753966    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08c6b2-6df7cb9cd-q9mx5"
Sep 11 17:39:18.754083 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:18.753971    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08c6b2-6df7cb9cd-q9mx5" event={"ID":"8d74b25a-7fd4-489c-943b-8fd444b32f38","Type":"ContainerDied","Data":"a7c36504d070c824123abc943d8e516098e8cef56f089f8e2ebd22fb4eb58a21"}
Sep 11 17:39:18.754083 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:18.754052    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08c6b2-6df7cb9cd-q9mx5" event={"ID":"8d74b25a-7fd4-489c-943b-8fd444b32f38","Type":"ContainerDied","Data":"9f11497f2f6b42202ce9ac5ea6714ccfa3be86022e20f6e5e0f9cfde35366555"}
Sep 11 17:39:18.761418 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:18.761306    2562 scope.go:117] "RemoveContainer" containerID="6988d1aeb216e89cc487af686998a28876712a9c1308f80e9b9e1d813bc3c5c0"
Sep 11 17:39:18.761884 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:39:18.761857    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6988d1aeb216e89cc487af686998a28876712a9c1308f80e9b9e1d813bc3c5c0\": container with ID starting with 6988d1aeb216e89cc487af686998a28876712a9c1308f80e9b9e1d813bc3c5c0 not found: ID does not exist" containerID="6988d1aeb216e89cc487af686998a28876712a9c1308f80e9b9e1d813bc3c5c0"
Sep 11 17:39:18.761978 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:18.761894    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6988d1aeb216e89cc487af686998a28876712a9c1308f80e9b9e1d813bc3c5c0"} err="failed to get container status \"6988d1aeb216e89cc487af686998a28876712a9c1308f80e9b9e1d813bc3c5c0\": rpc error: code = NotFound desc = could not find container \"6988d1aeb216e89cc487af686998a28876712a9c1308f80e9b9e1d813bc3c5c0\": container with ID starting with 6988d1aeb216e89cc487af686998a28876712a9c1308f80e9b9e1d813bc3c5c0 not found: ID does not exist"
Sep 11 17:39:18.761978 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:18.761919    2562 scope.go:117] "RemoveContainer" containerID="a7c36504d070c824123abc943d8e516098e8cef56f089f8e2ebd22fb4eb58a21"
Sep 11 17:39:18.772111 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:18.772084    2562 scope.go:117] "RemoveContainer" containerID="a7c36504d070c824123abc943d8e516098e8cef56f089f8e2ebd22fb4eb58a21"
Sep 11 17:39:18.772435 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:39:18.772407    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7c36504d070c824123abc943d8e516098e8cef56f089f8e2ebd22fb4eb58a21\": container with ID starting with a7c36504d070c824123abc943d8e516098e8cef56f089f8e2ebd22fb4eb58a21 not found: ID does not exist" containerID="a7c36504d070c824123abc943d8e516098e8cef56f089f8e2ebd22fb4eb58a21"
Sep 11 17:39:18.772516 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:18.772440    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7c36504d070c824123abc943d8e516098e8cef56f089f8e2ebd22fb4eb58a21"} err="failed to get container status \"a7c36504d070c824123abc943d8e516098e8cef56f089f8e2ebd22fb4eb58a21\": rpc error: code = NotFound desc = could not find container \"a7c36504d070c824123abc943d8e516098e8cef56f089f8e2ebd22fb4eb58a21\": container with ID starting with a7c36504d070c824123abc943d8e516098e8cef56f089f8e2ebd22fb4eb58a21 not found: ID does not exist"
Sep 11 17:39:18.829960 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:18.829935    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f249ed3-acb4-4e5d-b189-de21af9f1a25-openshift-service-ca\") pod \"5f249ed3-acb4-4e5d-b189-de21af9f1a25\" (UID: \"5f249ed3-acb4-4e5d-b189-de21af9f1a25\") "
Sep 11 17:39:18.830110 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:18.830022    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkw4l\" (UniqueName: \"kubernetes.io/projected/5f249ed3-acb4-4e5d-b189-de21af9f1a25-kube-api-access-tkw4l\") pod \"5f249ed3-acb4-4e5d-b189-de21af9f1a25\" (UID: \"5f249ed3-acb4-4e5d-b189-de21af9f1a25\") "
Sep 11 17:39:18.830110 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:18.830085    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gqzg\" (UniqueName: \"kubernetes.io/projected/8d74b25a-7fd4-489c-943b-8fd444b32f38-kube-api-access-8gqzg\") pod \"8d74b25a-7fd4-489c-943b-8fd444b32f38\" (UID: \"8d74b25a-7fd4-489c-943b-8fd444b32f38\") "
Sep 11 17:39:18.830238 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:18.830137    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f249ed3-acb4-4e5d-b189-de21af9f1a25-metrics-certs\") pod \"5f249ed3-acb4-4e5d-b189-de21af9f1a25\" (UID: \"5f249ed3-acb4-4e5d-b189-de21af9f1a25\") "
Sep 11 17:39:18.830238 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:18.830180    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f249ed3-acb4-4e5d-b189-de21af9f1a25-maas-api-tls\") pod \"5f249ed3-acb4-4e5d-b189-de21af9f1a25\" (UID: \"5f249ed3-acb4-4e5d-b189-de21af9f1a25\") "
Sep 11 17:39:18.830238 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:18.830215    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d74b25a-7fd4-489c-943b-8fd444b32f38-plugins-config-volume\") pod \"8d74b25a-7fd4-489c-943b-8fd444b32f38\" (UID: \"8d74b25a-7fd4-489c-943b-8fd444b32f38\") "
Sep 11 17:39:18.830400 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:18.830308    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f249ed3-acb4-4e5d-b189-de21af9f1a25-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5f249ed3-acb4-4e5d-b189-de21af9f1a25" (UID: "5f249ed3-acb4-4e5d-b189-de21af9f1a25"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:39:18.830640 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:18.830612    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d74b25a-7fd4-489c-943b-8fd444b32f38-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8d74b25a-7fd4-489c-943b-8fd444b32f38" (UID: "8d74b25a-7fd4-489c-943b-8fd444b32f38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:39:18.830770 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:18.830683    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f249ed3-acb4-4e5d-b189-de21af9f1a25-openshift-service-ca\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:39:18.832175 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:18.832155    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f249ed3-acb4-4e5d-b189-de21af9f1a25-kube-api-access-tkw4l" (OuterVolumeSpecName: "kube-api-access-tkw4l") pod "5f249ed3-acb4-4e5d-b189-de21af9f1a25" (UID: "5f249ed3-acb4-4e5d-b189-de21af9f1a25"). InnerVolumeSpecName "kube-api-access-tkw4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:39:18.832254 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:18.832184    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d74b25a-7fd4-489c-943b-8fd444b32f38-kube-api-access-8gqzg" (OuterVolumeSpecName: "kube-api-access-8gqzg") pod "8d74b25a-7fd4-489c-943b-8fd444b32f38" (UID: "8d74b25a-7fd4-489c-943b-8fd444b32f38"). InnerVolumeSpecName "kube-api-access-8gqzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:39:18.832608 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:18.832577    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f249ed3-acb4-4e5d-b189-de21af9f1a25-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5f249ed3-acb4-4e5d-b189-de21af9f1a25" (UID: "5f249ed3-acb4-4e5d-b189-de21af9f1a25"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:39:18.832698 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:18.832586    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f249ed3-acb4-4e5d-b189-de21af9f1a25-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5f249ed3-acb4-4e5d-b189-de21af9f1a25" (UID: "5f249ed3-acb4-4e5d-b189-de21af9f1a25"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:39:18.931376 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:18.931291    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gqzg\" (UniqueName: \"kubernetes.io/projected/8d74b25a-7fd4-489c-943b-8fd444b32f38-kube-api-access-8gqzg\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:39:18.931376 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:18.931324    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f249ed3-acb4-4e5d-b189-de21af9f1a25-metrics-certs\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:39:18.931376 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:18.931334    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f249ed3-acb4-4e5d-b189-de21af9f1a25-maas-api-tls\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:39:18.931376 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:18.931342    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d74b25a-7fd4-489c-943b-8fd444b32f38-plugins-config-volume\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:39:18.931376 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:18.931351    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkw4l\" (UniqueName: \"kubernetes.io/projected/5f249ed3-acb4-4e5d-b189-de21af9f1a25-kube-api-access-tkw4l\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:39:19.083026 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:19.082957    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w"]
Sep 11 17:39:19.090802 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:19.090774    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-08c6b2-977bd46f6-6kz2w"]
Sep 11 17:39:19.102368 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:19.102319    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-08c6b2-6df7cb9cd-q9mx5"]
Sep 11 17:39:19.105789 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:19.105733    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-08c6b2-6df7cb9cd-q9mx5"]
Sep 11 17:39:19.220897 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:19.220815    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-ac1a16/noten-model-ac1a16-kserve-78dc66fdcf-bx7m8"]
Sep 11 17:39:19.364580 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:19.364547    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"]
Sep 11 17:39:19.364960 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:19.364762    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh" podUID="cbc6aca4-5bd3-4520-b0d7-ba46df2890a4" containerName="istio-proxy" containerID="cri-o://e6514fd8444a43a9fc70bf58eef853d058ae76b01ef3fb2f9b17798f4ede1df7" gracePeriod=30
Sep 11 17:39:20.130280 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:20.130246    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f249ed3-acb4-4e5d-b189-de21af9f1a25" path="/var/lib/kubelet/pods/5f249ed3-acb4-4e5d-b189-de21af9f1a25/volumes"
Sep 11 17:39:20.130735 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:20.130717    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d74b25a-7fd4-489c-943b-8fd444b32f38" path="/var/lib/kubelet/pods/8d74b25a-7fd4-489c-943b-8fd444b32f38/volumes"
Sep 11 17:39:24.102215 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.102174    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-cbbb09/test-model-cbbb09-kserve-d4dcf7cb5-mrgdn" podUID="a287f58f-caa3-4b56-929b-faf88c022382" containerName="main" containerID="cri-o://34f8492f07d51c5dd15c27df537a758437ee7bf4b61f161ad924c06cc776daa4" gracePeriod=45
Sep 11 17:39:24.357942 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.357873    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cbbb09/test-model-cbbb09-kserve-d4dcf7cb5-mrgdn"
Sep 11 17:39:24.491475 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.491445    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a287f58f-caa3-4b56-929b-faf88c022382-tmp-dir\") pod \"a287f58f-caa3-4b56-929b-faf88c022382\" (UID: \"a287f58f-caa3-4b56-929b-faf88c022382\") "
Sep 11 17:39:24.491640 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.491493    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6brs4\" (UniqueName: \"kubernetes.io/projected/a287f58f-caa3-4b56-929b-faf88c022382-kube-api-access-6brs4\") pod \"a287f58f-caa3-4b56-929b-faf88c022382\" (UID: \"a287f58f-caa3-4b56-929b-faf88c022382\") "
Sep 11 17:39:24.491640 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.491529    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a287f58f-caa3-4b56-929b-faf88c022382-home\") pod \"a287f58f-caa3-4b56-929b-faf88c022382\" (UID: \"a287f58f-caa3-4b56-929b-faf88c022382\") "
Sep 11 17:39:24.491640 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.491565    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a287f58f-caa3-4b56-929b-faf88c022382-dshm\") pod \"a287f58f-caa3-4b56-929b-faf88c022382\" (UID: \"a287f58f-caa3-4b56-929b-faf88c022382\") "
Sep 11 17:39:24.491640 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.491590    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a287f58f-caa3-4b56-929b-faf88c022382-model-cache\") pod \"a287f58f-caa3-4b56-929b-faf88c022382\" (UID: \"a287f58f-caa3-4b56-929b-faf88c022382\") "
Sep 11 17:39:24.491874 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.491703    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a287f58f-caa3-4b56-929b-faf88c022382-tls-certs\") pod \"a287f58f-caa3-4b56-929b-faf88c022382\" (UID: \"a287f58f-caa3-4b56-929b-faf88c022382\") "
Sep 11 17:39:24.491874 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.491725    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a287f58f-caa3-4b56-929b-faf88c022382-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a287f58f-caa3-4b56-929b-faf88c022382" (UID: "a287f58f-caa3-4b56-929b-faf88c022382"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:39:24.491874 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.491804    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a287f58f-caa3-4b56-929b-faf88c022382-home" (OuterVolumeSpecName: "home") pod "a287f58f-caa3-4b56-929b-faf88c022382" (UID: "a287f58f-caa3-4b56-929b-faf88c022382"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:39:24.492020 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.491887    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a287f58f-caa3-4b56-929b-faf88c022382-model-cache" (OuterVolumeSpecName: "model-cache") pod "a287f58f-caa3-4b56-929b-faf88c022382" (UID: "a287f58f-caa3-4b56-929b-faf88c022382"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:39:24.492195 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.492165    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a287f58f-caa3-4b56-929b-faf88c022382-tmp-dir\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:39:24.492195 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.492193    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a287f58f-caa3-4b56-929b-faf88c022382-home\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:39:24.492361 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.492206    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a287f58f-caa3-4b56-929b-faf88c022382-model-cache\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:39:24.493698 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.493674    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a287f58f-caa3-4b56-929b-faf88c022382-dshm" (OuterVolumeSpecName: "dshm") pod "a287f58f-caa3-4b56-929b-faf88c022382" (UID: "a287f58f-caa3-4b56-929b-faf88c022382"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:39:24.493797 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.493777    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a287f58f-caa3-4b56-929b-faf88c022382-kube-api-access-6brs4" (OuterVolumeSpecName: "kube-api-access-6brs4") pod "a287f58f-caa3-4b56-929b-faf88c022382" (UID: "a287f58f-caa3-4b56-929b-faf88c022382"). InnerVolumeSpecName "kube-api-access-6brs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:39:24.493922 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.493904    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a287f58f-caa3-4b56-929b-faf88c022382-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a287f58f-caa3-4b56-929b-faf88c022382" (UID: "a287f58f-caa3-4b56-929b-faf88c022382"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:39:24.593633 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.593602    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6brs4\" (UniqueName: \"kubernetes.io/projected/a287f58f-caa3-4b56-929b-faf88c022382-kube-api-access-6brs4\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:39:24.593633 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.593631    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a287f58f-caa3-4b56-929b-faf88c022382-dshm\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:39:24.593829 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.593640    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a287f58f-caa3-4b56-929b-faf88c022382-tls-certs\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:39:24.608155 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.608085    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"
Sep 11 17:39:24.694794 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.694761    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wbbg\" (UniqueName: \"kubernetes.io/projected/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-kube-api-access-6wbbg\") pod \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\" (UID: \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\") "
Sep 11 17:39:24.695021 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.694807    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-istio-token\") pod \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\" (UID: \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\") "
Sep 11 17:39:24.695021 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.694832    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-workload-certs\") pod \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\" (UID: \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\") "
Sep 11 17:39:24.695021 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.694864    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-credential-socket\") pod \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\" (UID: \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\") "
Sep 11 17:39:24.695021 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.694895    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-istiod-ca-cert\") pod \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\" (UID: \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\") "
Sep 11 17:39:24.695021 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.694955    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-workload-socket\") pod \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\" (UID: \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\") "
Sep 11 17:39:24.695021 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.694985    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-istio-envoy\") pod \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\" (UID: \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\") "
Sep 11 17:39:24.695339 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.695116    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-istio-podinfo\") pod \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\" (UID: \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\") "
Sep 11 17:39:24.695339 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.695158    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-istio-data\") pod \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\" (UID: \"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4\") "
Sep 11 17:39:24.695339 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.695169    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cbc6aca4-5bd3-4520-b0d7-ba46df2890a4" (UID: "cbc6aca4-5bd3-4520-b0d7-ba46df2890a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:39:24.695339 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.695259    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cbc6aca4-5bd3-4520-b0d7-ba46df2890a4" (UID: "cbc6aca4-5bd3-4520-b0d7-ba46df2890a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:39:24.695546 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.695417    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cbc6aca4-5bd3-4520-b0d7-ba46df2890a4" (UID: "cbc6aca4-5bd3-4520-b0d7-ba46df2890a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:39:24.695546 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.695502    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-istio-data" (OuterVolumeSpecName: "istio-data") pod "cbc6aca4-5bd3-4520-b0d7-ba46df2890a4" (UID: "cbc6aca4-5bd3-4520-b0d7-ba46df2890a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:39:24.695652 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.695605    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cbc6aca4-5bd3-4520-b0d7-ba46df2890a4" (UID: "cbc6aca4-5bd3-4520-b0d7-ba46df2890a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:39:24.695652 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.695646    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-istio-data\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:39:24.695733 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.695663    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-workload-certs\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:39:24.695733 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.695678    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-istiod-ca-cert\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:39:24.695733 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.695691    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-workload-socket\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:39:24.697216 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.697192    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-kube-api-access-6wbbg" (OuterVolumeSpecName: "kube-api-access-6wbbg") pod "cbc6aca4-5bd3-4520-b0d7-ba46df2890a4" (UID: "cbc6aca4-5bd3-4520-b0d7-ba46df2890a4"). InnerVolumeSpecName "kube-api-access-6wbbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:39:24.697394 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.697371    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cbc6aca4-5bd3-4520-b0d7-ba46df2890a4" (UID: "cbc6aca4-5bd3-4520-b0d7-ba46df2890a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:39:24.697459 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.697445    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-istio-token" (OuterVolumeSpecName: "istio-token") pod "cbc6aca4-5bd3-4520-b0d7-ba46df2890a4" (UID: "cbc6aca4-5bd3-4520-b0d7-ba46df2890a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:39:24.697501 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.697475    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cbc6aca4-5bd3-4520-b0d7-ba46df2890a4" (UID: "cbc6aca4-5bd3-4520-b0d7-ba46df2890a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 17:39:24.787930 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.787899    2562 generic.go:356] "Generic (PLEG): container finished" podID="cbc6aca4-5bd3-4520-b0d7-ba46df2890a4" containerID="e6514fd8444a43a9fc70bf58eef853d058ae76b01ef3fb2f9b17798f4ede1df7" exitCode=0
Sep 11 17:39:24.788119 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.787949    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"
Sep 11 17:39:24.788119 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.787984    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh" event={"ID":"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4","Type":"ContainerDied","Data":"e6514fd8444a43a9fc70bf58eef853d058ae76b01ef3fb2f9b17798f4ede1df7"}
Sep 11 17:39:24.788119 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.788077    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh" event={"ID":"cbc6aca4-5bd3-4520-b0d7-ba46df2890a4","Type":"ContainerDied","Data":"965247c08ee5bf38fc1e32c3a4482e644baf1c7a5f4c48fe454de1cbea05199c"}
Sep 11 17:39:24.788119 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.788110    2562 scope.go:117] "RemoveContainer" containerID="e6514fd8444a43a9fc70bf58eef853d058ae76b01ef3fb2f9b17798f4ede1df7"
Sep 11 17:39:24.789591 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.789572    2562 generic.go:356] "Generic (PLEG): container finished" podID="a287f58f-caa3-4b56-929b-faf88c022382" containerID="34f8492f07d51c5dd15c27df537a758437ee7bf4b61f161ad924c06cc776daa4" exitCode=0
Sep 11 17:39:24.789685 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.789619    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cbbb09/test-model-cbbb09-kserve-d4dcf7cb5-mrgdn"
Sep 11 17:39:24.789685 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.789654    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cbbb09/test-model-cbbb09-kserve-d4dcf7cb5-mrgdn" event={"ID":"a287f58f-caa3-4b56-929b-faf88c022382","Type":"ContainerDied","Data":"34f8492f07d51c5dd15c27df537a758437ee7bf4b61f161ad924c06cc776daa4"}
Sep 11 17:39:24.789765 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.789692    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cbbb09/test-model-cbbb09-kserve-d4dcf7cb5-mrgdn" event={"ID":"a287f58f-caa3-4b56-929b-faf88c022382","Type":"ContainerDied","Data":"e5c4e9f90c0a87341aad065bd5a92768001ec1872ca310d2a508af9439f719df"}
Sep 11 17:39:24.796131 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.796111    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-istio-podinfo\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:39:24.796131 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.796132    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wbbg\" (UniqueName: \"kubernetes.io/projected/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-kube-api-access-6wbbg\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:39:24.796259 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.796141    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-istio-token\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:39:24.796259 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.796150    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-credential-socket\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:39:24.796259 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.796159    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4-istio-envoy\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:39:24.800715 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.800697    2562 scope.go:117] "RemoveContainer" containerID="e6514fd8444a43a9fc70bf58eef853d058ae76b01ef3fb2f9b17798f4ede1df7"
Sep 11 17:39:24.801030 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:39:24.801009    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6514fd8444a43a9fc70bf58eef853d058ae76b01ef3fb2f9b17798f4ede1df7\": container with ID starting with e6514fd8444a43a9fc70bf58eef853d058ae76b01ef3fb2f9b17798f4ede1df7 not found: ID does not exist" containerID="e6514fd8444a43a9fc70bf58eef853d058ae76b01ef3fb2f9b17798f4ede1df7"
Sep 11 17:39:24.801129 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.801041    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6514fd8444a43a9fc70bf58eef853d058ae76b01ef3fb2f9b17798f4ede1df7"} err="failed to get container status \"e6514fd8444a43a9fc70bf58eef853d058ae76b01ef3fb2f9b17798f4ede1df7\": rpc error: code = NotFound desc = could not find container \"e6514fd8444a43a9fc70bf58eef853d058ae76b01ef3fb2f9b17798f4ede1df7\": container with ID starting with e6514fd8444a43a9fc70bf58eef853d058ae76b01ef3fb2f9b17798f4ede1df7 not found: ID does not exist"
Sep 11 17:39:24.801129 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.801064    2562 scope.go:117] "RemoveContainer" containerID="34f8492f07d51c5dd15c27df537a758437ee7bf4b61f161ad924c06cc776daa4"
Sep 11 17:39:24.809642 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.809625    2562 scope.go:117] "RemoveContainer" containerID="34f8492f07d51c5dd15c27df537a758437ee7bf4b61f161ad924c06cc776daa4"
Sep 11 17:39:24.809900 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:39:24.809883    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34f8492f07d51c5dd15c27df537a758437ee7bf4b61f161ad924c06cc776daa4\": container with ID starting with 34f8492f07d51c5dd15c27df537a758437ee7bf4b61f161ad924c06cc776daa4 not found: ID does not exist" containerID="34f8492f07d51c5dd15c27df537a758437ee7bf4b61f161ad924c06cc776daa4"
Sep 11 17:39:24.809952 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.809905    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34f8492f07d51c5dd15c27df537a758437ee7bf4b61f161ad924c06cc776daa4"} err="failed to get container status \"34f8492f07d51c5dd15c27df537a758437ee7bf4b61f161ad924c06cc776daa4\": rpc error: code = NotFound desc = could not find container \"34f8492f07d51c5dd15c27df537a758437ee7bf4b61f161ad924c06cc776daa4\": container with ID starting with 34f8492f07d51c5dd15c27df537a758437ee7bf4b61f161ad924c06cc776daa4 not found: ID does not exist"
Sep 11 17:39:24.818523 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.818502    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"]
Sep 11 17:39:24.823766 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.823746    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-ac1a16-openshift-default-5fb49fb5bb-kldwh"]
Sep 11 17:39:24.833874 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.833855    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cbbb09/test-model-cbbb09-kserve-d4dcf7cb5-mrgdn"]
Sep 11 17:39:24.840579 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:24.840563    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-cbbb09/test-model-cbbb09-kserve-d4dcf7cb5-mrgdn"]
Sep 11 17:39:26.126729 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:26.126695    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a287f58f-caa3-4b56-929b-faf88c022382" path="/var/lib/kubelet/pods/a287f58f-caa3-4b56-929b-faf88c022382/volumes"
Sep 11 17:39:26.127148 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:26.127134    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbc6aca4-5bd3-4520-b0d7-ba46df2890a4" path="/var/lib/kubelet/pods/cbc6aca4-5bd3-4520-b0d7-ba46df2890a4/volumes"
Sep 11 17:39:34.240106 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:34.240006    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-ac1a16/noten-model-ac1a16-kserve-78dc66fdcf-bx7m8" podUID="bc3d82f7-197f-47e6-8cd0-d229c3fd18fd" containerName="main" containerID="cri-o://489ab4ae7f440a4e7cf2d1fa78bdb19a89e77f5aa8dea9a5c8428737626aa2e6" gracePeriod=45
Sep 11 17:39:34.487693 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:34.487666    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ac1a16/noten-model-ac1a16-kserve-78dc66fdcf-bx7m8"
Sep 11 17:39:34.592025 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:34.591977    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc3d82f7-197f-47e6-8cd0-d229c3fd18fd-model-cache\") pod \"bc3d82f7-197f-47e6-8cd0-d229c3fd18fd\" (UID: \"bc3d82f7-197f-47e6-8cd0-d229c3fd18fd\") "
Sep 11 17:39:34.592225 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:34.592098    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7z56h\" (UniqueName: \"kubernetes.io/projected/bc3d82f7-197f-47e6-8cd0-d229c3fd18fd-kube-api-access-7z56h\") pod \"bc3d82f7-197f-47e6-8cd0-d229c3fd18fd\" (UID: \"bc3d82f7-197f-47e6-8cd0-d229c3fd18fd\") "
Sep 11 17:39:34.592225 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:34.592144    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc3d82f7-197f-47e6-8cd0-d229c3fd18fd-home\") pod \"bc3d82f7-197f-47e6-8cd0-d229c3fd18fd\" (UID: \"bc3d82f7-197f-47e6-8cd0-d229c3fd18fd\") "
Sep 11 17:39:34.592225 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:34.592167    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc3d82f7-197f-47e6-8cd0-d229c3fd18fd-tls-certs\") pod \"bc3d82f7-197f-47e6-8cd0-d229c3fd18fd\" (UID: \"bc3d82f7-197f-47e6-8cd0-d229c3fd18fd\") "
Sep 11 17:39:34.592390 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:34.592233    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc3d82f7-197f-47e6-8cd0-d229c3fd18fd-tmp-dir\") pod \"bc3d82f7-197f-47e6-8cd0-d229c3fd18fd\" (UID: \"bc3d82f7-197f-47e6-8cd0-d229c3fd18fd\") "
Sep 11 17:39:34.592390 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:34.592260    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc3d82f7-197f-47e6-8cd0-d229c3fd18fd-dshm\") pod \"bc3d82f7-197f-47e6-8cd0-d229c3fd18fd\" (UID: \"bc3d82f7-197f-47e6-8cd0-d229c3fd18fd\") "
Sep 11 17:39:34.592390 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:34.592257    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc3d82f7-197f-47e6-8cd0-d229c3fd18fd-model-cache" (OuterVolumeSpecName: "model-cache") pod "bc3d82f7-197f-47e6-8cd0-d229c3fd18fd" (UID: "bc3d82f7-197f-47e6-8cd0-d229c3fd18fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:39:34.592390 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:34.592350    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc3d82f7-197f-47e6-8cd0-d229c3fd18fd-home" (OuterVolumeSpecName: "home") pod "bc3d82f7-197f-47e6-8cd0-d229c3fd18fd" (UID: "bc3d82f7-197f-47e6-8cd0-d229c3fd18fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:39:34.592583 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:34.592494    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc3d82f7-197f-47e6-8cd0-d229c3fd18fd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bc3d82f7-197f-47e6-8cd0-d229c3fd18fd" (UID: "bc3d82f7-197f-47e6-8cd0-d229c3fd18fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:39:34.592637 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:34.592596    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc3d82f7-197f-47e6-8cd0-d229c3fd18fd-home\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:39:34.592689 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:34.592640    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc3d82f7-197f-47e6-8cd0-d229c3fd18fd-tmp-dir\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:39:34.592689 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:34.592657    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc3d82f7-197f-47e6-8cd0-d229c3fd18fd-model-cache\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:39:34.594355 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:34.594325    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc3d82f7-197f-47e6-8cd0-d229c3fd18fd-kube-api-access-7z56h" (OuterVolumeSpecName: "kube-api-access-7z56h") pod "bc3d82f7-197f-47e6-8cd0-d229c3fd18fd" (UID: "bc3d82f7-197f-47e6-8cd0-d229c3fd18fd"). InnerVolumeSpecName "kube-api-access-7z56h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:39:34.594738 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:34.594709    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc3d82f7-197f-47e6-8cd0-d229c3fd18fd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bc3d82f7-197f-47e6-8cd0-d229c3fd18fd" (UID: "bc3d82f7-197f-47e6-8cd0-d229c3fd18fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:39:34.594738 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:34.594725    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc3d82f7-197f-47e6-8cd0-d229c3fd18fd-dshm" (OuterVolumeSpecName: "dshm") pod "bc3d82f7-197f-47e6-8cd0-d229c3fd18fd" (UID: "bc3d82f7-197f-47e6-8cd0-d229c3fd18fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:39:34.693495 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:34.693458    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc3d82f7-197f-47e6-8cd0-d229c3fd18fd-dshm\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:39:34.693495 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:34.693488    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z56h\" (UniqueName: \"kubernetes.io/projected/bc3d82f7-197f-47e6-8cd0-d229c3fd18fd-kube-api-access-7z56h\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:39:34.693495 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:34.693499    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc3d82f7-197f-47e6-8cd0-d229c3fd18fd-tls-certs\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:39:34.843387 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:34.843298    2562 generic.go:356] "Generic (PLEG): container finished" podID="bc3d82f7-197f-47e6-8cd0-d229c3fd18fd" containerID="489ab4ae7f440a4e7cf2d1fa78bdb19a89e77f5aa8dea9a5c8428737626aa2e6" exitCode=0
Sep 11 17:39:34.843387 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:34.843339    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ac1a16/noten-model-ac1a16-kserve-78dc66fdcf-bx7m8"
Sep 11 17:39:34.843561 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:34.843382    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ac1a16/noten-model-ac1a16-kserve-78dc66fdcf-bx7m8" event={"ID":"bc3d82f7-197f-47e6-8cd0-d229c3fd18fd","Type":"ContainerDied","Data":"489ab4ae7f440a4e7cf2d1fa78bdb19a89e77f5aa8dea9a5c8428737626aa2e6"}
Sep 11 17:39:34.843561 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:34.843422    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ac1a16/noten-model-ac1a16-kserve-78dc66fdcf-bx7m8" event={"ID":"bc3d82f7-197f-47e6-8cd0-d229c3fd18fd","Type":"ContainerDied","Data":"3095d9997558d05be67e540826e92193354d1b25b802e4e18526381087751b27"}
Sep 11 17:39:34.843561 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:34.843440    2562 scope.go:117] "RemoveContainer" containerID="489ab4ae7f440a4e7cf2d1fa78bdb19a89e77f5aa8dea9a5c8428737626aa2e6"
Sep 11 17:39:34.852307 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:34.852290    2562 scope.go:117] "RemoveContainer" containerID="489ab4ae7f440a4e7cf2d1fa78bdb19a89e77f5aa8dea9a5c8428737626aa2e6"
Sep 11 17:39:34.852573 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:39:34.852555    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"489ab4ae7f440a4e7cf2d1fa78bdb19a89e77f5aa8dea9a5c8428737626aa2e6\": container with ID starting with 489ab4ae7f440a4e7cf2d1fa78bdb19a89e77f5aa8dea9a5c8428737626aa2e6 not found: ID does not exist" containerID="489ab4ae7f440a4e7cf2d1fa78bdb19a89e77f5aa8dea9a5c8428737626aa2e6"
Sep 11 17:39:34.852642 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:34.852586    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"489ab4ae7f440a4e7cf2d1fa78bdb19a89e77f5aa8dea9a5c8428737626aa2e6"} err="failed to get container status \"489ab4ae7f440a4e7cf2d1fa78bdb19a89e77f5aa8dea9a5c8428737626aa2e6\": rpc error: code = NotFound desc = could not find container \"489ab4ae7f440a4e7cf2d1fa78bdb19a89e77f5aa8dea9a5c8428737626aa2e6\": container with ID starting with 489ab4ae7f440a4e7cf2d1fa78bdb19a89e77f5aa8dea9a5c8428737626aa2e6 not found: ID does not exist"
Sep 11 17:39:34.865514 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:34.865492    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-ac1a16/noten-model-ac1a16-kserve-78dc66fdcf-bx7m8"]
Sep 11 17:39:34.869242 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:34.869222    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-ac1a16/noten-model-ac1a16-kserve-78dc66fdcf-bx7m8"]
Sep 11 17:39:36.126302 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:36.126268    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc3d82f7-197f-47e6-8cd0-d229c3fd18fd" path="/var/lib/kubelet/pods/bc3d82f7-197f-47e6-8cd0-d229c3fd18fd/volumes"
Sep 11 17:39:36.172577 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:36.172537    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf"]
Sep 11 17:39:36.173021 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:36.172803    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf" podUID="a57696c0-abc7-4258-896c-476008734dfe" containerName="maas-api" containerID="cri-o://b50f654e0f8f614a73cecfa57777e75be4dbe8f2451b7feb11f518846a2adcc5" gracePeriod=30
Sep 11 17:39:36.433849 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:36.433824    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf"
Sep 11 17:39:36.511611 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:36.511560    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsx89\" (UniqueName: \"kubernetes.io/projected/a57696c0-abc7-4258-896c-476008734dfe-kube-api-access-hsx89\") pod \"a57696c0-abc7-4258-896c-476008734dfe\" (UID: \"a57696c0-abc7-4258-896c-476008734dfe\") "
Sep 11 17:39:36.511611 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:36.511619    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a57696c0-abc7-4258-896c-476008734dfe-metrics-certs\") pod \"a57696c0-abc7-4258-896c-476008734dfe\" (UID: \"a57696c0-abc7-4258-896c-476008734dfe\") "
Sep 11 17:39:36.511845 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:36.511739    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a57696c0-abc7-4258-896c-476008734dfe-openshift-service-ca\") pod \"a57696c0-abc7-4258-896c-476008734dfe\" (UID: \"a57696c0-abc7-4258-896c-476008734dfe\") "
Sep 11 17:39:36.511845 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:36.511831    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a57696c0-abc7-4258-896c-476008734dfe-maas-api-tls\") pod \"a57696c0-abc7-4258-896c-476008734dfe\" (UID: \"a57696c0-abc7-4258-896c-476008734dfe\") "
Sep 11 17:39:36.512063 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:36.511973    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a57696c0-abc7-4258-896c-476008734dfe-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a57696c0-abc7-4258-896c-476008734dfe" (UID: "a57696c0-abc7-4258-896c-476008734dfe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:39:36.512196 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:36.512176    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a57696c0-abc7-4258-896c-476008734dfe-openshift-service-ca\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:39:36.513859 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:36.513832    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a57696c0-abc7-4258-896c-476008734dfe-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a57696c0-abc7-4258-896c-476008734dfe" (UID: "a57696c0-abc7-4258-896c-476008734dfe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:39:36.513969 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:36.513878    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a57696c0-abc7-4258-896c-476008734dfe-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a57696c0-abc7-4258-896c-476008734dfe" (UID: "a57696c0-abc7-4258-896c-476008734dfe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:39:36.513969 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:36.513913    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a57696c0-abc7-4258-896c-476008734dfe-kube-api-access-hsx89" (OuterVolumeSpecName: "kube-api-access-hsx89") pod "a57696c0-abc7-4258-896c-476008734dfe" (UID: "a57696c0-abc7-4258-896c-476008734dfe"). InnerVolumeSpecName "kube-api-access-hsx89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:39:36.613815 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:36.613732    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a57696c0-abc7-4258-896c-476008734dfe-maas-api-tls\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:39:36.613815 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:36.613776    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsx89\" (UniqueName: \"kubernetes.io/projected/a57696c0-abc7-4258-896c-476008734dfe-kube-api-access-hsx89\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:39:36.613815 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:36.613791    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a57696c0-abc7-4258-896c-476008734dfe-metrics-certs\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:39:36.856535 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:36.856499    2562 generic.go:356] "Generic (PLEG): container finished" podID="a57696c0-abc7-4258-896c-476008734dfe" containerID="b50f654e0f8f614a73cecfa57777e75be4dbe8f2451b7feb11f518846a2adcc5" exitCode=0
Sep 11 17:39:36.856693 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:36.856546    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf"
Sep 11 17:39:36.856693 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:36.856584    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf" event={"ID":"a57696c0-abc7-4258-896c-476008734dfe","Type":"ContainerDied","Data":"b50f654e0f8f614a73cecfa57777e75be4dbe8f2451b7feb11f518846a2adcc5"}
Sep 11 17:39:36.856693 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:36.856639    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf" event={"ID":"a57696c0-abc7-4258-896c-476008734dfe","Type":"ContainerDied","Data":"dd5d55eb8fbba23c103c1abdf073eb9450fb63e7a512342d300c83d4eaeca397"}
Sep 11 17:39:36.856693 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:36.856671    2562 scope.go:117] "RemoveContainer" containerID="b50f654e0f8f614a73cecfa57777e75be4dbe8f2451b7feb11f518846a2adcc5"
Sep 11 17:39:36.866166 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:36.866148    2562 scope.go:117] "RemoveContainer" containerID="b50f654e0f8f614a73cecfa57777e75be4dbe8f2451b7feb11f518846a2adcc5"
Sep 11 17:39:36.866438 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:39:36.866417    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b50f654e0f8f614a73cecfa57777e75be4dbe8f2451b7feb11f518846a2adcc5\": container with ID starting with b50f654e0f8f614a73cecfa57777e75be4dbe8f2451b7feb11f518846a2adcc5 not found: ID does not exist" containerID="b50f654e0f8f614a73cecfa57777e75be4dbe8f2451b7feb11f518846a2adcc5"
Sep 11 17:39:36.866503 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:36.866452    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b50f654e0f8f614a73cecfa57777e75be4dbe8f2451b7feb11f518846a2adcc5"} err="failed to get container status \"b50f654e0f8f614a73cecfa57777e75be4dbe8f2451b7feb11f518846a2adcc5\": rpc error: code = NotFound desc = could not find container \"b50f654e0f8f614a73cecfa57777e75be4dbe8f2451b7feb11f518846a2adcc5\": container with ID starting with b50f654e0f8f614a73cecfa57777e75be4dbe8f2451b7feb11f518846a2adcc5 not found: ID does not exist"
Sep 11 17:39:36.883063 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:36.883026    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf"]
Sep 11 17:39:36.885770 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:36.885738    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cbbb09-678cf86f9b-d8njf"]
Sep 11 17:39:38.125585 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:38.125550    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a57696c0-abc7-4258-896c-476008734dfe" path="/var/lib/kubelet/pods/a57696c0-abc7-4258-896c-476008734dfe/volumes"
Sep 11 17:39:43.194018 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.193957    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-c7d192/distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9"]
Sep 11 17:39:43.194423 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.194373    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a57696c0-abc7-4258-896c-476008734dfe" containerName="maas-api"
Sep 11 17:39:43.194423 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.194386    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a57696c0-abc7-4258-896c-476008734dfe" containerName="maas-api"
Sep 11 17:39:43.194423 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.194397    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbc6aca4-5bd3-4520-b0d7-ba46df2890a4" containerName="istio-proxy"
Sep 11 17:39:43.194423 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.194403    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbc6aca4-5bd3-4520-b0d7-ba46df2890a4" containerName="istio-proxy"
Sep 11 17:39:43.194423 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.194414    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f249ed3-acb4-4e5d-b189-de21af9f1a25" containerName="maas-api"
Sep 11 17:39:43.194423 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.194421    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f249ed3-acb4-4e5d-b189-de21af9f1a25" containerName="maas-api"
Sep 11 17:39:43.194611 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.194435    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d74b25a-7fd4-489c-943b-8fd444b32f38" containerName="payload-pre-processing"
Sep 11 17:39:43.194611 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.194445    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d74b25a-7fd4-489c-943b-8fd444b32f38" containerName="payload-pre-processing"
Sep 11 17:39:43.194611 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.194453    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc3d82f7-197f-47e6-8cd0-d229c3fd18fd" containerName="main"
Sep 11 17:39:43.194611 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.194458    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc3d82f7-197f-47e6-8cd0-d229c3fd18fd" containerName="main"
Sep 11 17:39:43.194611 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.194469    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a287f58f-caa3-4b56-929b-faf88c022382" containerName="main"
Sep 11 17:39:43.194611 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.194474    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a287f58f-caa3-4b56-929b-faf88c022382" containerName="main"
Sep 11 17:39:43.194611 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.194550    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a57696c0-abc7-4258-896c-476008734dfe" containerName="maas-api"
Sep 11 17:39:43.194611 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.194560    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a287f58f-caa3-4b56-929b-faf88c022382" containerName="main"
Sep 11 17:39:43.194611 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.194566    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d74b25a-7fd4-489c-943b-8fd444b32f38" containerName="payload-pre-processing"
Sep 11 17:39:43.194611 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.194573    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbc6aca4-5bd3-4520-b0d7-ba46df2890a4" containerName="istio-proxy"
Sep 11 17:39:43.194611 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.194579    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc3d82f7-197f-47e6-8cd0-d229c3fd18fd" containerName="main"
Sep 11 17:39:43.194611 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.194585    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f249ed3-acb4-4e5d-b189-de21af9f1a25" containerName="maas-api"
Sep 11 17:39:43.199095 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.199076    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c7d192/distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9"
Sep 11 17:39:43.201370 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.201343    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c7d192\"/\"openshift-service-ca.crt\""
Sep 11 17:39:43.201515 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.201380    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c7d192\"/\"kube-root-ca.crt\""
Sep 11 17:39:43.202083 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.202061    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c7d192\"/\"default-dockercfg-jxfg5\""
Sep 11 17:39:43.202173 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.202137    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c7d192\"/\"distinct-2-w0-c7d192-kserve-self-signed-certs\""
Sep 11 17:39:43.207757 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.207734    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c7d192/distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9"]
Sep 11 17:39:43.277074 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.277037    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcef6d19-ab56-4c36-8725-ce2253241999-dshm\") pod \"distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9\" (UID: \"fcef6d19-ab56-4c36-8725-ce2253241999\") " pod="e2e-models-e2e-worker-w0-c7d192/distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9"
Sep 11 17:39:43.277263 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.277083    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcef6d19-ab56-4c36-8725-ce2253241999-model-cache\") pod \"distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9\" (UID: \"fcef6d19-ab56-4c36-8725-ce2253241999\") " pod="e2e-models-e2e-worker-w0-c7d192/distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9"
Sep 11 17:39:43.277263 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.277208    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcef6d19-ab56-4c36-8725-ce2253241999-tmp-dir\") pod \"distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9\" (UID: \"fcef6d19-ab56-4c36-8725-ce2253241999\") " pod="e2e-models-e2e-worker-w0-c7d192/distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9"
Sep 11 17:39:43.277263 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.277240    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcef6d19-ab56-4c36-8725-ce2253241999-home\") pod \"distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9\" (UID: \"fcef6d19-ab56-4c36-8725-ce2253241999\") " pod="e2e-models-e2e-worker-w0-c7d192/distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9"
Sep 11 17:39:43.277438 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.277262    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d78vz\" (UniqueName: \"kubernetes.io/projected/fcef6d19-ab56-4c36-8725-ce2253241999-kube-api-access-d78vz\") pod \"distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9\" (UID: \"fcef6d19-ab56-4c36-8725-ce2253241999\") " pod="e2e-models-e2e-worker-w0-c7d192/distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9"
Sep 11 17:39:43.277438 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.277339    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcef6d19-ab56-4c36-8725-ce2253241999-tls-certs\") pod \"distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9\" (UID: \"fcef6d19-ab56-4c36-8725-ce2253241999\") " pod="e2e-models-e2e-worker-w0-c7d192/distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9"
Sep 11 17:39:43.378194 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.378159    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcef6d19-ab56-4c36-8725-ce2253241999-tls-certs\") pod \"distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9\" (UID: \"fcef6d19-ab56-4c36-8725-ce2253241999\") " pod="e2e-models-e2e-worker-w0-c7d192/distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9"
Sep 11 17:39:43.378379 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.378225    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcef6d19-ab56-4c36-8725-ce2253241999-dshm\") pod \"distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9\" (UID: \"fcef6d19-ab56-4c36-8725-ce2253241999\") " pod="e2e-models-e2e-worker-w0-c7d192/distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9"
Sep 11 17:39:43.378379 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.378262    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcef6d19-ab56-4c36-8725-ce2253241999-model-cache\") pod \"distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9\" (UID: \"fcef6d19-ab56-4c36-8725-ce2253241999\") " pod="e2e-models-e2e-worker-w0-c7d192/distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9"
Sep 11 17:39:43.378379 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.378322    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcef6d19-ab56-4c36-8725-ce2253241999-tmp-dir\") pod \"distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9\" (UID: \"fcef6d19-ab56-4c36-8725-ce2253241999\") " pod="e2e-models-e2e-worker-w0-c7d192/distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9"
Sep 11 17:39:43.378379 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.378343    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcef6d19-ab56-4c36-8725-ce2253241999-home\") pod \"distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9\" (UID: \"fcef6d19-ab56-4c36-8725-ce2253241999\") " pod="e2e-models-e2e-worker-w0-c7d192/distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9"
Sep 11 17:39:43.378379 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.378361    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d78vz\" (UniqueName: \"kubernetes.io/projected/fcef6d19-ab56-4c36-8725-ce2253241999-kube-api-access-d78vz\") pod \"distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9\" (UID: \"fcef6d19-ab56-4c36-8725-ce2253241999\") " pod="e2e-models-e2e-worker-w0-c7d192/distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9"
Sep 11 17:39:43.378736 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.378705    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcef6d19-ab56-4c36-8725-ce2253241999-model-cache\") pod \"distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9\" (UID: \"fcef6d19-ab56-4c36-8725-ce2253241999\") " pod="e2e-models-e2e-worker-w0-c7d192/distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9"
Sep 11 17:39:43.378874 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.378767    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcef6d19-ab56-4c36-8725-ce2253241999-tmp-dir\") pod \"distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9\" (UID: \"fcef6d19-ab56-4c36-8725-ce2253241999\") " pod="e2e-models-e2e-worker-w0-c7d192/distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9"
Sep 11 17:39:43.378874 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.378790    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcef6d19-ab56-4c36-8725-ce2253241999-home\") pod \"distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9\" (UID: \"fcef6d19-ab56-4c36-8725-ce2253241999\") " pod="e2e-models-e2e-worker-w0-c7d192/distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9"
Sep 11 17:39:43.380478 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.380455    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcef6d19-ab56-4c36-8725-ce2253241999-dshm\") pod \"distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9\" (UID: \"fcef6d19-ab56-4c36-8725-ce2253241999\") " pod="e2e-models-e2e-worker-w0-c7d192/distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9"
Sep 11 17:39:43.380792 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.380774    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcef6d19-ab56-4c36-8725-ce2253241999-tls-certs\") pod \"distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9\" (UID: \"fcef6d19-ab56-4c36-8725-ce2253241999\") " pod="e2e-models-e2e-worker-w0-c7d192/distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9"
Sep 11 17:39:43.386553 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.386531    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d78vz\" (UniqueName: \"kubernetes.io/projected/fcef6d19-ab56-4c36-8725-ce2253241999-kube-api-access-d78vz\") pod \"distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9\" (UID: \"fcef6d19-ab56-4c36-8725-ce2253241999\") " pod="e2e-models-e2e-worker-w0-c7d192/distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9"
Sep 11 17:39:43.508715 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.508623    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c7d192/distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9"
Sep 11 17:39:43.789164 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.789139    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c7d192/distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9"]
Sep 11 17:39:43.791469 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:39:43.791433    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfcef6d19_ab56_4c36_8725_ce2253241999.slice/crio-c763d6e554c8f2a34ab726c721e729596bfba9e0cf632a8ffcfd005350b6f629 WatchSource:0}: Error finding container c763d6e554c8f2a34ab726c721e729596bfba9e0cf632a8ffcfd005350b6f629: Status 404 returned error can't find the container with id c763d6e554c8f2a34ab726c721e729596bfba9e0cf632a8ffcfd005350b6f629
Sep 11 17:39:43.893489 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.893457    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c7d192/distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9" event={"ID":"fcef6d19-ab56-4c36-8725-ce2253241999","Type":"ContainerStarted","Data":"b341462a694a05c501a5f46d870a8587fb8e62074219669923f4e2165d773abd"}
Sep 11 17:39:43.893489 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.893496    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c7d192/distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9" event={"ID":"fcef6d19-ab56-4c36-8725-ce2253241999","Type":"ContainerStarted","Data":"c763d6e554c8f2a34ab726c721e729596bfba9e0cf632a8ffcfd005350b6f629"}
Sep 11 17:39:43.909597 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:43.909533    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-c7d192/distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9" podStartSLOduration=0.909511107 podStartE2EDuration="909.511107ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:39:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:39:43.908974909 +0000 UTC m=+1264.302892376" watchObservedRunningTime="2026-09-11 17:39:43.909511107 +0000 UTC m=+1264.303428581"
Sep 11 17:39:44.442259 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:44.442225    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"]
Sep 11 17:39:44.445950 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:44.445924    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"
Sep 11 17:39:44.448647 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:44.448622    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-2fac54-openshift-default-dockercfg-2wzjg\""
Sep 11 17:39:44.462960 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:44.462938    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"]
Sep 11 17:39:44.493171 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:44.493142    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-workload-certs\") pod \"e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk\" (UID: \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\") " pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"
Sep 11 17:39:44.493171 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:44.493175    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-istio-token\") pod \"e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk\" (UID: \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\") " pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"
Sep 11 17:39:44.493370 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:44.493290    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-istiod-ca-cert\") pod \"e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk\" (UID: \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\") " pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"
Sep 11 17:39:44.493370 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:44.493352    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-istio-podinfo\") pod \"e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk\" (UID: \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\") " pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"
Sep 11 17:39:44.493443 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:44.493404    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-istio-envoy\") pod \"e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk\" (UID: \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\") " pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"
Sep 11 17:39:44.493476 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:44.493463    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlrnb\" (UniqueName: \"kubernetes.io/projected/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-kube-api-access-zlrnb\") pod \"e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk\" (UID: \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\") " pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"
Sep 11 17:39:44.493517 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:44.493493    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-istio-data\") pod \"e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk\" (UID: \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\") " pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"
Sep 11 17:39:44.493560 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:44.493548    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-credential-socket\") pod \"e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk\" (UID: \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\") " pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"
Sep 11 17:39:44.493602 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:44.493587    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-workload-socket\") pod \"e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk\" (UID: \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\") " pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"
Sep 11 17:39:44.594960 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:44.594920    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-istio-envoy\") pod \"e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk\" (UID: \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\") " pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"
Sep 11 17:39:44.594960 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:44.594963    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlrnb\" (UniqueName: \"kubernetes.io/projected/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-kube-api-access-zlrnb\") pod \"e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk\" (UID: \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\") " pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"
Sep 11 17:39:44.595224 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:44.594985    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-istio-data\") pod \"e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk\" (UID: \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\") " pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"
Sep 11 17:39:44.595224 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:44.595057    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-credential-socket\") pod \"e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk\" (UID: \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\") " pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"
Sep 11 17:39:44.595224 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:44.595089    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-workload-socket\") pod \"e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk\" (UID: \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\") " pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"
Sep 11 17:39:44.595466 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:44.595214    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-workload-certs\") pod \"e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk\" (UID: \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\") " pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"
Sep 11 17:39:44.595466 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:44.595260    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-istio-token\") pod \"e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk\" (UID: \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\") " pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"
Sep 11 17:39:44.595466 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:44.595341    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-istiod-ca-cert\") pod \"e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk\" (UID: \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\") " pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"
Sep 11 17:39:44.595466 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:44.595387    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-istio-podinfo\") pod \"e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk\" (UID: \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\") " pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"
Sep 11 17:39:44.595677 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:44.595480    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-istio-data\") pod \"e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk\" (UID: \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\") " pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"
Sep 11 17:39:44.595677 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:44.595561    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-credential-socket\") pod \"e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk\" (UID: \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\") " pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"
Sep 11 17:39:44.595677 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:44.595592    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-workload-socket\") pod \"e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk\" (UID: \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\") " pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"
Sep 11 17:39:44.595838 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:44.595678    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-workload-certs\") pod \"e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk\" (UID: \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\") " pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"
Sep 11 17:39:44.596090 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:44.596071    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-istiod-ca-cert\") pod \"e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk\" (UID: \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\") " pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"
Sep 11 17:39:44.597434 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:44.597409    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-istio-envoy\") pod \"e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk\" (UID: \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\") " pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"
Sep 11 17:39:44.597699 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:44.597683    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-istio-podinfo\") pod \"e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk\" (UID: \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\") " pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"
Sep 11 17:39:44.603399 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:44.603369    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-istio-token\") pod \"e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk\" (UID: \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\") " pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"
Sep 11 17:39:44.603773 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:44.603757    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlrnb\" (UniqueName: \"kubernetes.io/projected/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-kube-api-access-zlrnb\") pod \"e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk\" (UID: \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\") " pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"
Sep 11 17:39:44.756927 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:44.756830    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"
Sep 11 17:39:45.034499 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:45.034472    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"]
Sep 11 17:39:45.035606 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:39:45.035581    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2faa8d27_577e_4dba_8d81_e5cd42ba7e02.slice/crio-401443cfd840afeb3134e47dea52d15466bcb8dbf20bd50a8443a727a7dcbe0a WatchSource:0}: Error finding container 401443cfd840afeb3134e47dea52d15466bcb8dbf20bd50a8443a727a7dcbe0a: Status 404 returned error can't find the container with id 401443cfd840afeb3134e47dea52d15466bcb8dbf20bd50a8443a727a7dcbe0a
Sep 11 17:39:45.037962 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:45.037925    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:39:45.038067 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:45.038020    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:39:45.038067 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:45.038053    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:39:45.912749 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:45.912713    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk" event={"ID":"2faa8d27-577e-4dba-8d81-e5cd42ba7e02","Type":"ContainerStarted","Data":"e28076078c10dc6c00a3c8706d941f72b23e8f75cf9f17f83d41b0175cc506c2"}
Sep 11 17:39:45.913159 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:45.912771    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk" event={"ID":"2faa8d27-577e-4dba-8d81-e5cd42ba7e02","Type":"ContainerStarted","Data":"401443cfd840afeb3134e47dea52d15466bcb8dbf20bd50a8443a727a7dcbe0a"}
Sep 11 17:39:45.933128 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:45.933076    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk" podStartSLOduration=1.9330614289999999 podStartE2EDuration="1.933061429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:39:45.930967985 +0000 UTC m=+1266.324885435" watchObservedRunningTime="2026-09-11 17:39:45.933061429 +0000 UTC m=+1266.326978883"
Sep 11 17:39:46.757644 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:46.757607    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"
Sep 11 17:39:46.763143 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:46.763115    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"
Sep 11 17:39:46.917682 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:46.917652    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"
Sep 11 17:39:46.918728 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:46.918707    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"
Sep 11 17:39:53.219732 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:53.219696    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fac54-7dfb59d654-ntv7d"]
Sep 11 17:39:53.223534 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:53.223517    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fac54-7dfb59d654-ntv7d"
Sep 11 17:39:53.227599 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:53.227578    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-2fac54\""
Sep 11 17:39:53.228455 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:53.228441    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-2fac54-dockercfg-hc2pw\""
Sep 11 17:39:53.241371 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:53.241341    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fac54-7dfb59d654-ntv7d"]
Sep 11 17:39:53.285429 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:53.285385    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c23adc-e901-4cc9-80d1-18e632314638-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2fac54-7dfb59d654-ntv7d\" (UID: \"84c23adc-e901-4cc9-80d1-18e632314638\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fac54-7dfb59d654-ntv7d"
Sep 11 17:39:53.285643 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:53.285459    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ctvm\" (UniqueName: \"kubernetes.io/projected/84c23adc-e901-4cc9-80d1-18e632314638-kube-api-access-8ctvm\") pod \"payload-pre-processing-e2e-shared-a-w3-2fac54-7dfb59d654-ntv7d\" (UID: \"84c23adc-e901-4cc9-80d1-18e632314638\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fac54-7dfb59d654-ntv7d"
Sep 11 17:39:53.386827 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:53.386788    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c23adc-e901-4cc9-80d1-18e632314638-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2fac54-7dfb59d654-ntv7d\" (UID: \"84c23adc-e901-4cc9-80d1-18e632314638\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fac54-7dfb59d654-ntv7d"
Sep 11 17:39:53.387052 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:53.386854    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ctvm\" (UniqueName: \"kubernetes.io/projected/84c23adc-e901-4cc9-80d1-18e632314638-kube-api-access-8ctvm\") pod \"payload-pre-processing-e2e-shared-a-w3-2fac54-7dfb59d654-ntv7d\" (UID: \"84c23adc-e901-4cc9-80d1-18e632314638\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fac54-7dfb59d654-ntv7d"
Sep 11 17:39:53.387787 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:53.387763    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c23adc-e901-4cc9-80d1-18e632314638-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2fac54-7dfb59d654-ntv7d\" (UID: \"84c23adc-e901-4cc9-80d1-18e632314638\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fac54-7dfb59d654-ntv7d"
Sep 11 17:39:53.401562 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:53.401533    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ctvm\" (UniqueName: \"kubernetes.io/projected/84c23adc-e901-4cc9-80d1-18e632314638-kube-api-access-8ctvm\") pod \"payload-pre-processing-e2e-shared-a-w3-2fac54-7dfb59d654-ntv7d\" (UID: \"84c23adc-e901-4cc9-80d1-18e632314638\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fac54-7dfb59d654-ntv7d"
Sep 11 17:39:53.509042 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:53.508910    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c7d192/distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9"
Sep 11 17:39:53.509042 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:53.508972    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-c7d192/distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9"
Sep 11 17:39:53.521640 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:53.521613    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-c7d192/distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9"
Sep 11 17:39:53.534533 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:53.534502    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fac54-7dfb59d654-ntv7d"
Sep 11 17:39:53.805711 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:53.805682    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fac54-7dfb59d654-ntv7d"]
Sep 11 17:39:53.807729 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:39:53.807701    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c23adc_e901_4cc9_80d1_18e632314638.slice/crio-a6fadad520915b4bb1c997d3ef6bbcc2c803bad987a0cccd8df70d969c9575be WatchSource:0}: Error finding container a6fadad520915b4bb1c997d3ef6bbcc2c803bad987a0cccd8df70d969c9575be: Status 404 returned error can't find the container with id a6fadad520915b4bb1c997d3ef6bbcc2c803bad987a0cccd8df70d969c9575be
Sep 11 17:39:53.954585 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:53.954552    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fac54-7dfb59d654-ntv7d" event={"ID":"84c23adc-e901-4cc9-80d1-18e632314638","Type":"ContainerStarted","Data":"cba98995bebc03fe0d20990aad09c959be4be2f2a08caa74a423eff7d7cd4820"}
Sep 11 17:39:53.954765 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:53.954608    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fac54-7dfb59d654-ntv7d" event={"ID":"84c23adc-e901-4cc9-80d1-18e632314638","Type":"ContainerStarted","Data":"a6fadad520915b4bb1c997d3ef6bbcc2c803bad987a0cccd8df70d969c9575be"}
Sep 11 17:39:53.954765 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:53.954640    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fac54-7dfb59d654-ntv7d"
Sep 11 17:39:53.966316 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:53.966283    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c7d192/distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9"
Sep 11 17:39:53.971588 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:39:53.971535    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fac54-7dfb59d654-ntv7d" podStartSLOduration=0.971512868 podStartE2EDuration="971.512868ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:39:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:39:53.970052998 +0000 UTC m=+1274.363970452" watchObservedRunningTime="2026-09-11 17:39:53.971512868 +0000 UTC m=+1274.365430325"
Sep 11 17:40:04.966227 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:04.966181    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fac54-7dfb59d654-ntv7d"
Sep 11 17:40:12.059541 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:12.059435    2562 generic.go:356] "Generic (PLEG): container finished" podID="2faa8d27-577e-4dba-8d81-e5cd42ba7e02" containerID="e28076078c10dc6c00a3c8706d941f72b23e8f75cf9f17f83d41b0175cc506c2" exitCode=0
Sep 11 17:40:12.059541 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:12.059511    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk" event={"ID":"2faa8d27-577e-4dba-8d81-e5cd42ba7e02","Type":"ContainerDied","Data":"e28076078c10dc6c00a3c8706d941f72b23e8f75cf9f17f83d41b0175cc506c2"}
Sep 11 17:40:12.060007 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:12.059939    2562 scope.go:117] "RemoveContainer" containerID="e28076078c10dc6c00a3c8706d941f72b23e8f75cf9f17f83d41b0175cc506c2"
Sep 11 17:40:12.061120 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:12.061089    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:40:12.061240 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:12.061171    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:40:12.061240 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:12.061218    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:40:13.069708 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:13.069672    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk" event={"ID":"2faa8d27-577e-4dba-8d81-e5cd42ba7e02","Type":"ContainerStarted","Data":"0e8e3d6f522aaaf49aa9221950ab4684a3fd7b3eb81e15bbe8b92ee8f461254d"}
Sep 11 17:40:13.758667 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:13.758621    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"
Sep 11 17:40:13.758884 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:13.758793    2562 patch_prober.go:28] interesting pod/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 11 17:40:13.758964 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:13.758881    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk" podUID="2faa8d27-577e-4dba-8d81-e5cd42ba7e02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 11 17:40:14.757636 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:14.757588    2562 patch_prober.go:28] interesting pod/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 11 17:40:14.758185 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:14.757668    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk" podUID="2faa8d27-577e-4dba-8d81-e5cd42ba7e02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 11 17:40:15.757663 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:15.757621    2562 patch_prober.go:28] interesting pod/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 11 17:40:15.758314 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:15.758281    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk" podUID="2faa8d27-577e-4dba-8d81-e5cd42ba7e02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 11 17:40:16.757687 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:16.757647    2562 patch_prober.go:28] interesting pod/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 11 17:40:16.758222 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:16.757727    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk" podUID="2faa8d27-577e-4dba-8d81-e5cd42ba7e02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 11 17:40:16.917902 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:16.917870    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"
Sep 11 17:40:17.757984 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:17.757937    2562 patch_prober.go:28] interesting pod/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 11 17:40:17.758469 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:17.758025    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk" podUID="2faa8d27-577e-4dba-8d81-e5cd42ba7e02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 11 17:40:18.567551 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:18.567510    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"]
Sep 11 17:40:18.570376 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:18.570354    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"
Sep 11 17:40:18.574044 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:18.574017    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-6fc9e6-openshift-default-dockercfg-9zkkw\""
Sep 11 17:40:18.591014 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:18.590964    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"]
Sep 11 17:40:18.642113 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:18.642078    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7868f970-d989-409a-bf67-f184f8e39ca0-istio-podinfo\") pod \"e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5\" (UID: \"7868f970-d989-409a-bf67-f184f8e39ca0\") " pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"
Sep 11 17:40:18.642113 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:18.642116    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7868f970-d989-409a-bf67-f184f8e39ca0-workload-certs\") pod \"e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5\" (UID: \"7868f970-d989-409a-bf67-f184f8e39ca0\") " pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"
Sep 11 17:40:18.642329 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:18.642133    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7868f970-d989-409a-bf67-f184f8e39ca0-credential-socket\") pod \"e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5\" (UID: \"7868f970-d989-409a-bf67-f184f8e39ca0\") " pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"
Sep 11 17:40:18.642329 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:18.642208    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7868f970-d989-409a-bf67-f184f8e39ca0-istio-data\") pod \"e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5\" (UID: \"7868f970-d989-409a-bf67-f184f8e39ca0\") " pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"
Sep 11 17:40:18.642329 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:18.642229    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7868f970-d989-409a-bf67-f184f8e39ca0-istiod-ca-cert\") pod \"e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5\" (UID: \"7868f970-d989-409a-bf67-f184f8e39ca0\") " pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"
Sep 11 17:40:18.642329 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:18.642303    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7868f970-d989-409a-bf67-f184f8e39ca0-istio-token\") pod \"e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5\" (UID: \"7868f970-d989-409a-bf67-f184f8e39ca0\") " pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"
Sep 11 17:40:18.642466 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:18.642379    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7868f970-d989-409a-bf67-f184f8e39ca0-workload-socket\") pod \"e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5\" (UID: \"7868f970-d989-409a-bf67-f184f8e39ca0\") " pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"
Sep 11 17:40:18.642466 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:18.642421    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7868f970-d989-409a-bf67-f184f8e39ca0-istio-envoy\") pod \"e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5\" (UID: \"7868f970-d989-409a-bf67-f184f8e39ca0\") " pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"
Sep 11 17:40:18.642466 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:18.642451    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdlrs\" (UniqueName: \"kubernetes.io/projected/7868f970-d989-409a-bf67-f184f8e39ca0-kube-api-access-wdlrs\") pod \"e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5\" (UID: \"7868f970-d989-409a-bf67-f184f8e39ca0\") " pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"
Sep 11 17:40:18.743464 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:18.743422    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7868f970-d989-409a-bf67-f184f8e39ca0-workload-socket\") pod \"e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5\" (UID: \"7868f970-d989-409a-bf67-f184f8e39ca0\") " pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"
Sep 11 17:40:18.743464 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:18.743460    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7868f970-d989-409a-bf67-f184f8e39ca0-istio-envoy\") pod \"e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5\" (UID: \"7868f970-d989-409a-bf67-f184f8e39ca0\") " pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"
Sep 11 17:40:18.743731 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:18.743479    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdlrs\" (UniqueName: \"kubernetes.io/projected/7868f970-d989-409a-bf67-f184f8e39ca0-kube-api-access-wdlrs\") pod \"e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5\" (UID: \"7868f970-d989-409a-bf67-f184f8e39ca0\") " pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"
Sep 11 17:40:18.743731 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:18.743540    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7868f970-d989-409a-bf67-f184f8e39ca0-istio-podinfo\") pod \"e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5\" (UID: \"7868f970-d989-409a-bf67-f184f8e39ca0\") " pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"
Sep 11 17:40:18.743731 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:18.743558    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7868f970-d989-409a-bf67-f184f8e39ca0-workload-certs\") pod \"e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5\" (UID: \"7868f970-d989-409a-bf67-f184f8e39ca0\") " pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"
Sep 11 17:40:18.743731 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:18.743575    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7868f970-d989-409a-bf67-f184f8e39ca0-credential-socket\") pod \"e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5\" (UID: \"7868f970-d989-409a-bf67-f184f8e39ca0\") " pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"
Sep 11 17:40:18.743731 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:18.743618    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7868f970-d989-409a-bf67-f184f8e39ca0-istio-data\") pod \"e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5\" (UID: \"7868f970-d989-409a-bf67-f184f8e39ca0\") " pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"
Sep 11 17:40:18.743731 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:18.743642    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7868f970-d989-409a-bf67-f184f8e39ca0-istiod-ca-cert\") pod \"e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5\" (UID: \"7868f970-d989-409a-bf67-f184f8e39ca0\") " pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"
Sep 11 17:40:18.743731 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:18.743679    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7868f970-d989-409a-bf67-f184f8e39ca0-istio-token\") pod \"e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5\" (UID: \"7868f970-d989-409a-bf67-f184f8e39ca0\") " pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"
Sep 11 17:40:18.744144 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:18.743916    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7868f970-d989-409a-bf67-f184f8e39ca0-workload-socket\") pod \"e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5\" (UID: \"7868f970-d989-409a-bf67-f184f8e39ca0\") " pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"
Sep 11 17:40:18.744144 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:18.744042    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7868f970-d989-409a-bf67-f184f8e39ca0-workload-certs\") pod \"e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5\" (UID: \"7868f970-d989-409a-bf67-f184f8e39ca0\") " pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"
Sep 11 17:40:18.744144 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:18.744116    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7868f970-d989-409a-bf67-f184f8e39ca0-credential-socket\") pod \"e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5\" (UID: \"7868f970-d989-409a-bf67-f184f8e39ca0\") " pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"
Sep 11 17:40:18.744308 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:18.744223    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7868f970-d989-409a-bf67-f184f8e39ca0-istio-data\") pod \"e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5\" (UID: \"7868f970-d989-409a-bf67-f184f8e39ca0\") " pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"
Sep 11 17:40:18.744686 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:18.744658    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7868f970-d989-409a-bf67-f184f8e39ca0-istiod-ca-cert\") pod \"e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5\" (UID: \"7868f970-d989-409a-bf67-f184f8e39ca0\") " pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"
Sep 11 17:40:18.745960 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:18.745941    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7868f970-d989-409a-bf67-f184f8e39ca0-istio-envoy\") pod \"e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5\" (UID: \"7868f970-d989-409a-bf67-f184f8e39ca0\") " pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"
Sep 11 17:40:18.746114 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:18.746099    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7868f970-d989-409a-bf67-f184f8e39ca0-istio-podinfo\") pod \"e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5\" (UID: \"7868f970-d989-409a-bf67-f184f8e39ca0\") " pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"
Sep 11 17:40:18.753824 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:18.753802    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7868f970-d989-409a-bf67-f184f8e39ca0-istio-token\") pod \"e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5\" (UID: \"7868f970-d989-409a-bf67-f184f8e39ca0\") " pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"
Sep 11 17:40:18.753983 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:18.753967    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdlrs\" (UniqueName: \"kubernetes.io/projected/7868f970-d989-409a-bf67-f184f8e39ca0-kube-api-access-wdlrs\") pod \"e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5\" (UID: \"7868f970-d989-409a-bf67-f184f8e39ca0\") " pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"
Sep 11 17:40:18.762149 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:18.762129    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"
Sep 11 17:40:18.763187 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:18.763169    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"
Sep 11 17:40:18.882334 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:18.882302    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"
Sep 11 17:40:19.171844 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:19.171816    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"]
Sep 11 17:40:19.173535 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:40:19.173506    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7868f970_d989_409a_bf67_f184f8e39ca0.slice/crio-d5aa06ebf46d42f65fe315e078d1a2c9b7f8eaa50647d039ad222978fc9921ca WatchSource:0}: Error finding container d5aa06ebf46d42f65fe315e078d1a2c9b7f8eaa50647d039ad222978fc9921ca: Status 404 returned error can't find the container with id d5aa06ebf46d42f65fe315e078d1a2c9b7f8eaa50647d039ad222978fc9921ca
Sep 11 17:40:19.176047 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:19.176006    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:40:19.176131 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:19.176105    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:40:19.176175 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:19.176147    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:40:20.109515 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:20.109478    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5" event={"ID":"7868f970-d989-409a-bf67-f184f8e39ca0","Type":"ContainerStarted","Data":"c361f1de4c44da0554bcfdcc08dd2351e1ed4efe418d5be2c10d1d25e6409a09"}
Sep 11 17:40:20.109931 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:20.109537    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5" event={"ID":"7868f970-d989-409a-bf67-f184f8e39ca0","Type":"ContainerStarted","Data":"d5aa06ebf46d42f65fe315e078d1a2c9b7f8eaa50647d039ad222978fc9921ca"}
Sep 11 17:40:20.166081 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:20.166015    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5" podStartSLOduration=2.16598057 podStartE2EDuration="2.16598057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:40:20.164765768 +0000 UTC m=+1300.558683224" watchObservedRunningTime="2026-09-11 17:40:20.16598057 +0000 UTC m=+1300.559898026"
Sep 11 17:40:20.883039 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:20.882987    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"
Sep 11 17:40:20.888344 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:20.888319    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"
Sep 11 17:40:21.114851 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:21.114801    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"
Sep 11 17:40:21.115841 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:21.115821    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"
Sep 11 17:40:28.041711 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:28.041678    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6fc9e6-c8c75bccf-b7vlq"]
Sep 11 17:40:28.054072 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:28.054040    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6fc9e6-c8c75bccf-b7vlq"
Sep 11 17:40:28.056733 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:28.056705    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-6fc9e6-dockercfg-9pxzh\""
Sep 11 17:40:28.056876 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:28.056718    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-6fc9e6\""
Sep 11 17:40:28.057217 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:28.057195    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6fc9e6-c8c75bccf-b7vlq"]
Sep 11 17:40:28.141340 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:28.140330    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66qsx\" (UniqueName: \"kubernetes.io/projected/b5db0334-d05c-4df6-9ccf-b833171869af-kube-api-access-66qsx\") pod \"payload-processing-e2e-shared-b-w3-6fc9e6-c8c75bccf-b7vlq\" (UID: \"b5db0334-d05c-4df6-9ccf-b833171869af\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6fc9e6-c8c75bccf-b7vlq"
Sep 11 17:40:28.141340 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:28.140443    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5db0334-d05c-4df6-9ccf-b833171869af-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-6fc9e6-c8c75bccf-b7vlq\" (UID: \"b5db0334-d05c-4df6-9ccf-b833171869af\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6fc9e6-c8c75bccf-b7vlq"
Sep 11 17:40:28.241637 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:28.241600    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5db0334-d05c-4df6-9ccf-b833171869af-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-6fc9e6-c8c75bccf-b7vlq\" (UID: \"b5db0334-d05c-4df6-9ccf-b833171869af\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6fc9e6-c8c75bccf-b7vlq"
Sep 11 17:40:28.241863 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:28.241759    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66qsx\" (UniqueName: \"kubernetes.io/projected/b5db0334-d05c-4df6-9ccf-b833171869af-kube-api-access-66qsx\") pod \"payload-processing-e2e-shared-b-w3-6fc9e6-c8c75bccf-b7vlq\" (UID: \"b5db0334-d05c-4df6-9ccf-b833171869af\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6fc9e6-c8c75bccf-b7vlq"
Sep 11 17:40:28.242341 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:28.242319    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5db0334-d05c-4df6-9ccf-b833171869af-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-6fc9e6-c8c75bccf-b7vlq\" (UID: \"b5db0334-d05c-4df6-9ccf-b833171869af\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6fc9e6-c8c75bccf-b7vlq"
Sep 11 17:40:28.250105 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:28.250076    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66qsx\" (UniqueName: \"kubernetes.io/projected/b5db0334-d05c-4df6-9ccf-b833171869af-kube-api-access-66qsx\") pod \"payload-processing-e2e-shared-b-w3-6fc9e6-c8c75bccf-b7vlq\" (UID: \"b5db0334-d05c-4df6-9ccf-b833171869af\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6fc9e6-c8c75bccf-b7vlq"
Sep 11 17:40:28.366179 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:28.366148    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6fc9e6-c8c75bccf-b7vlq"
Sep 11 17:40:28.674648 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:28.674620    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6fc9e6-c8c75bccf-b7vlq"]
Sep 11 17:40:28.676372 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:40:28.676343    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5db0334_d05c_4df6_9ccf_b833171869af.slice/crio-a076ad0437a2a2b4b50f9c719c425065c9bda2fed8029f4aeddf277ebc265d5b WatchSource:0}: Error finding container a076ad0437a2a2b4b50f9c719c425065c9bda2fed8029f4aeddf277ebc265d5b: Status 404 returned error can't find the container with id a076ad0437a2a2b4b50f9c719c425065c9bda2fed8029f4aeddf277ebc265d5b
Sep 11 17:40:29.158191 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:29.158149    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6fc9e6-c8c75bccf-b7vlq" event={"ID":"b5db0334-d05c-4df6-9ccf-b833171869af","Type":"ContainerStarted","Data":"8cad24b0987a2db41718cdfc70b9ee6ef91a2699a312a5ba59579960db18abd5"}
Sep 11 17:40:29.158620 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:29.158219    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6fc9e6-c8c75bccf-b7vlq" event={"ID":"b5db0334-d05c-4df6-9ccf-b833171869af","Type":"ContainerStarted","Data":"a076ad0437a2a2b4b50f9c719c425065c9bda2fed8029f4aeddf277ebc265d5b"}
Sep 11 17:40:29.158620 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:29.158349    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6fc9e6-c8c75bccf-b7vlq"
Sep 11 17:40:29.179062 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:29.178985    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6fc9e6-c8c75bccf-b7vlq" podStartSLOduration=1.178967731 podStartE2EDuration="1.178967731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:40:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:40:29.177308926 +0000 UTC m=+1309.571226381" watchObservedRunningTime="2026-09-11 17:40:29.178967731 +0000 UTC m=+1309.572885185"
Sep 11 17:40:40.165618 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:40.165543    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6fc9e6-c8c75bccf-b7vlq"
Sep 11 17:40:43.304346 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:43.304306    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6709f1a7-55c86c99d-gfvl6"]
Sep 11 17:40:43.304895 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:43.304823    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-6709f1a7-55c86c99d-gfvl6" podUID="67502d7f-4095-4544-8561-231e3b23ceac" containerName="payload-pre-processing" containerID="cri-o://bf2cffd0db5c4249587cb51d46b331ff1155948baaaf57f8467147812d71cae2" gracePeriod=30
Sep 11 17:40:43.641534 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:43.641505    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6709f1a7-55c86c99d-gfvl6"
Sep 11 17:40:43.804852 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:43.804804    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdqfs\" (UniqueName: \"kubernetes.io/projected/67502d7f-4095-4544-8561-231e3b23ceac-kube-api-access-fdqfs\") pod \"67502d7f-4095-4544-8561-231e3b23ceac\" (UID: \"67502d7f-4095-4544-8561-231e3b23ceac\") "
Sep 11 17:40:43.805092 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:43.804923    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67502d7f-4095-4544-8561-231e3b23ceac-plugins-config-volume\") pod \"67502d7f-4095-4544-8561-231e3b23ceac\" (UID: \"67502d7f-4095-4544-8561-231e3b23ceac\") "
Sep 11 17:40:43.805375 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:43.805343    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67502d7f-4095-4544-8561-231e3b23ceac-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "67502d7f-4095-4544-8561-231e3b23ceac" (UID: "67502d7f-4095-4544-8561-231e3b23ceac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:40:43.806939 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:43.806916    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67502d7f-4095-4544-8561-231e3b23ceac-kube-api-access-fdqfs" (OuterVolumeSpecName: "kube-api-access-fdqfs") pod "67502d7f-4095-4544-8561-231e3b23ceac" (UID: "67502d7f-4095-4544-8561-231e3b23ceac"). InnerVolumeSpecName "kube-api-access-fdqfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:40:43.906505 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:43.906469    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67502d7f-4095-4544-8561-231e3b23ceac-plugins-config-volume\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:40:43.906505 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:43.906499    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdqfs\" (UniqueName: \"kubernetes.io/projected/67502d7f-4095-4544-8561-231e3b23ceac-kube-api-access-fdqfs\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:40:44.239342 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:44.239308    2562 generic.go:356] "Generic (PLEG): container finished" podID="67502d7f-4095-4544-8561-231e3b23ceac" containerID="bf2cffd0db5c4249587cb51d46b331ff1155948baaaf57f8467147812d71cae2" exitCode=0
Sep 11 17:40:44.239540 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:44.239354    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6709f1a7-55c86c99d-gfvl6"
Sep 11 17:40:44.239540 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:44.239404    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6709f1a7-55c86c99d-gfvl6" event={"ID":"67502d7f-4095-4544-8561-231e3b23ceac","Type":"ContainerDied","Data":"bf2cffd0db5c4249587cb51d46b331ff1155948baaaf57f8467147812d71cae2"}
Sep 11 17:40:44.239540 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:44.239465    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6709f1a7-55c86c99d-gfvl6" event={"ID":"67502d7f-4095-4544-8561-231e3b23ceac","Type":"ContainerDied","Data":"4c0d1178762cd2face10932de6acf3ceacf54faa9aa2cd7f88574587b16d5f6d"}
Sep 11 17:40:44.239540 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:44.239486    2562 scope.go:117] "RemoveContainer" containerID="bf2cffd0db5c4249587cb51d46b331ff1155948baaaf57f8467147812d71cae2"
Sep 11 17:40:44.250340 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:44.250318    2562 scope.go:117] "RemoveContainer" containerID="bf2cffd0db5c4249587cb51d46b331ff1155948baaaf57f8467147812d71cae2"
Sep 11 17:40:44.250625 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:40:44.250607    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf2cffd0db5c4249587cb51d46b331ff1155948baaaf57f8467147812d71cae2\": container with ID starting with bf2cffd0db5c4249587cb51d46b331ff1155948baaaf57f8467147812d71cae2 not found: ID does not exist" containerID="bf2cffd0db5c4249587cb51d46b331ff1155948baaaf57f8467147812d71cae2"
Sep 11 17:40:44.250677 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:44.250633    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf2cffd0db5c4249587cb51d46b331ff1155948baaaf57f8467147812d71cae2"} err="failed to get container status \"bf2cffd0db5c4249587cb51d46b331ff1155948baaaf57f8467147812d71cae2\": rpc error: code = NotFound desc = could not find container \"bf2cffd0db5c4249587cb51d46b331ff1155948baaaf57f8467147812d71cae2\": container with ID starting with bf2cffd0db5c4249587cb51d46b331ff1155948baaaf57f8467147812d71cae2 not found: ID does not exist"
Sep 11 17:40:44.259917 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:44.259886    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6709f1a7-55c86c99d-gfvl6"]
Sep 11 17:40:44.264176 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:44.264152    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6709f1a7-55c86c99d-gfvl6"]
Sep 11 17:40:46.128023 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:46.127963    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67502d7f-4095-4544-8561-231e3b23ceac" path="/var/lib/kubelet/pods/67502d7f-4095-4544-8561-231e3b23ceac/volumes"
Sep 11 17:40:47.260299 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:47.260193    2562 generic.go:356] "Generic (PLEG): container finished" podID="7868f970-d989-409a-bf67-f184f8e39ca0" containerID="c361f1de4c44da0554bcfdcc08dd2351e1ed4efe418d5be2c10d1d25e6409a09" exitCode=0
Sep 11 17:40:47.261107 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:47.260864    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5" event={"ID":"7868f970-d989-409a-bf67-f184f8e39ca0","Type":"ContainerDied","Data":"c361f1de4c44da0554bcfdcc08dd2351e1ed4efe418d5be2c10d1d25e6409a09"}
Sep 11 17:40:47.262183 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:47.262162    2562 scope.go:117] "RemoveContainer" containerID="c361f1de4c44da0554bcfdcc08dd2351e1ed4efe418d5be2c10d1d25e6409a09"
Sep 11 17:40:47.264798 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:47.264359    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:40:47.264798 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:47.264456    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:40:47.264798 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:47.264506    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:40:48.270744 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:48.270709    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5" event={"ID":"7868f970-d989-409a-bf67-f184f8e39ca0","Type":"ContainerStarted","Data":"2d70a58ff0d680f67455c7fd24e0f4507061bfd3f17043e28db2f6fd88eae4d4"}
Sep 11 17:40:48.883513 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:48.883476    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"
Sep 11 17:40:48.884902 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:48.884874    2562 patch_prober.go:28] interesting pod/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Sep 11 17:40:48.885057 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:48.884940    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5" podUID="7868f970-d989-409a-bf67-f184f8e39ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Sep 11 17:40:49.883583 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:49.883516    2562 patch_prober.go:28] interesting pod/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Sep 11 17:40:49.884193 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:49.884144    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5" podUID="7868f970-d989-409a-bf67-f184f8e39ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Sep 11 17:40:50.883477 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:50.883433    2562 patch_prober.go:28] interesting pod/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Sep 11 17:40:50.883684 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:50.883505    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5" podUID="7868f970-d989-409a-bf67-f184f8e39ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Sep 11 17:40:51.115085 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:51.115048    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"
Sep 11 17:40:51.882722 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:51.882684    2562 patch_prober.go:28] interesting pod/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Sep 11 17:40:51.882913 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:51.882757    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5" podUID="7868f970-d989-409a-bf67-f184f8e39ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Sep 11 17:40:52.882894 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:52.882853    2562 patch_prober.go:28] interesting pod/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Sep 11 17:40:52.883423 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:52.882936    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5" podUID="7868f970-d989-409a-bf67-f184f8e39ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Sep 11 17:40:53.882718 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:53.882682    2562 patch_prober.go:28] interesting pod/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Sep 11 17:40:53.882899 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:53.882751    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5" podUID="7868f970-d989-409a-bf67-f184f8e39ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Sep 11 17:40:54.887274 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:54.887243    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"
Sep 11 17:40:54.888410 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:40:54.888392    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"
Sep 11 17:41:34.847849 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:34.847811    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"]
Sep 11 17:41:34.848371 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:34.848080    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql" podUID="e5eab92b-01a6-4f8e-a118-04d2db7fb047" containerName="istio-proxy" containerID="cri-o://1a1d1a0f8ef94520bda9310bd172742ffccf798b3937086da000db2ff6bb63c1" gracePeriod=30
Sep 11 17:41:40.108044 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.108015    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"
Sep 11 17:41:40.201846 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.201736    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5eab92b-01a6-4f8e-a118-04d2db7fb047-workload-socket\") pod \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\" (UID: \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\") "
Sep 11 17:41:40.201846 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.201795    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5eab92b-01a6-4f8e-a118-04d2db7fb047-istio-data\") pod \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\" (UID: \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\") "
Sep 11 17:41:40.201846 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.201824    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5eab92b-01a6-4f8e-a118-04d2db7fb047-workload-certs\") pod \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\" (UID: \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\") "
Sep 11 17:41:40.202198 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.201864    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5eab92b-01a6-4f8e-a118-04d2db7fb047-istio-podinfo\") pod \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\" (UID: \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\") "
Sep 11 17:41:40.202198 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.201896    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5eab92b-01a6-4f8e-a118-04d2db7fb047-istio-token\") pod \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\" (UID: \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\") "
Sep 11 17:41:40.202198 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.201916    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5eab92b-01a6-4f8e-a118-04d2db7fb047-credential-socket\") pod \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\" (UID: \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\") "
Sep 11 17:41:40.202198 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.202031    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5eab92b-01a6-4f8e-a118-04d2db7fb047-istiod-ca-cert\") pod \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\" (UID: \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\") "
Sep 11 17:41:40.202198 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.202059    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gd75p\" (UniqueName: \"kubernetes.io/projected/e5eab92b-01a6-4f8e-a118-04d2db7fb047-kube-api-access-gd75p\") pod \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\" (UID: \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\") "
Sep 11 17:41:40.202198 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.202096    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5eab92b-01a6-4f8e-a118-04d2db7fb047-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e5eab92b-01a6-4f8e-a118-04d2db7fb047" (UID: "e5eab92b-01a6-4f8e-a118-04d2db7fb047"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:41:40.202198 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.202142    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5eab92b-01a6-4f8e-a118-04d2db7fb047-istio-data" (OuterVolumeSpecName: "istio-data") pod "e5eab92b-01a6-4f8e-a118-04d2db7fb047" (UID: "e5eab92b-01a6-4f8e-a118-04d2db7fb047"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:41:40.202198 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.202168    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5eab92b-01a6-4f8e-a118-04d2db7fb047-istio-envoy\") pod \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\" (UID: \"e5eab92b-01a6-4f8e-a118-04d2db7fb047\") "
Sep 11 17:41:40.202603 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.202258    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5eab92b-01a6-4f8e-a118-04d2db7fb047-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e5eab92b-01a6-4f8e-a118-04d2db7fb047" (UID: "e5eab92b-01a6-4f8e-a118-04d2db7fb047"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:41:40.202603 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.202325    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5eab92b-01a6-4f8e-a118-04d2db7fb047-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e5eab92b-01a6-4f8e-a118-04d2db7fb047" (UID: "e5eab92b-01a6-4f8e-a118-04d2db7fb047"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:41:40.202603 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.202399    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5eab92b-01a6-4f8e-a118-04d2db7fb047-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e5eab92b-01a6-4f8e-a118-04d2db7fb047" (UID: "e5eab92b-01a6-4f8e-a118-04d2db7fb047"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:41:40.202603 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.202565    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5eab92b-01a6-4f8e-a118-04d2db7fb047-credential-socket\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:41:40.202603 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.202585    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5eab92b-01a6-4f8e-a118-04d2db7fb047-istiod-ca-cert\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:41:40.202603 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.202600    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5eab92b-01a6-4f8e-a118-04d2db7fb047-workload-socket\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:41:40.202842 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.202612    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5eab92b-01a6-4f8e-a118-04d2db7fb047-istio-data\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:41:40.202842 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.202625    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5eab92b-01a6-4f8e-a118-04d2db7fb047-workload-certs\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:41:40.204269 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.204246    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5eab92b-01a6-4f8e-a118-04d2db7fb047-istio-token" (OuterVolumeSpecName: "istio-token") pod "e5eab92b-01a6-4f8e-a118-04d2db7fb047" (UID: "e5eab92b-01a6-4f8e-a118-04d2db7fb047"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:41:40.204362 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.204259    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5eab92b-01a6-4f8e-a118-04d2db7fb047-kube-api-access-gd75p" (OuterVolumeSpecName: "kube-api-access-gd75p") pod "e5eab92b-01a6-4f8e-a118-04d2db7fb047" (UID: "e5eab92b-01a6-4f8e-a118-04d2db7fb047"). InnerVolumeSpecName "kube-api-access-gd75p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:41:40.204589 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.204570    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5eab92b-01a6-4f8e-a118-04d2db7fb047-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e5eab92b-01a6-4f8e-a118-04d2db7fb047" (UID: "e5eab92b-01a6-4f8e-a118-04d2db7fb047"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:41:40.204657 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.204581    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5eab92b-01a6-4f8e-a118-04d2db7fb047-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e5eab92b-01a6-4f8e-a118-04d2db7fb047" (UID: "e5eab92b-01a6-4f8e-a118-04d2db7fb047"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 17:41:40.303504 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.303465    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd75p\" (UniqueName: \"kubernetes.io/projected/e5eab92b-01a6-4f8e-a118-04d2db7fb047-kube-api-access-gd75p\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:41:40.303504 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.303502    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5eab92b-01a6-4f8e-a118-04d2db7fb047-istio-envoy\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:41:40.303504 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.303512    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5eab92b-01a6-4f8e-a118-04d2db7fb047-istio-podinfo\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:41:40.303739 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.303520    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5eab92b-01a6-4f8e-a118-04d2db7fb047-istio-token\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:41:40.551318 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.551231    2562 generic.go:356] "Generic (PLEG): container finished" podID="e5eab92b-01a6-4f8e-a118-04d2db7fb047" containerID="1a1d1a0f8ef94520bda9310bd172742ffccf798b3937086da000db2ff6bb63c1" exitCode=0
Sep 11 17:41:40.551318 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.551282    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"
Sep 11 17:41:40.551506 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.551314    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql" event={"ID":"e5eab92b-01a6-4f8e-a118-04d2db7fb047","Type":"ContainerDied","Data":"1a1d1a0f8ef94520bda9310bd172742ffccf798b3937086da000db2ff6bb63c1"}
Sep 11 17:41:40.551506 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.551381    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql" event={"ID":"e5eab92b-01a6-4f8e-a118-04d2db7fb047","Type":"ContainerDied","Data":"32f439f4f107e0c15c1fa595446bf4ff7f8fb6e636a261c6dd810e9a5e186153"}
Sep 11 17:41:40.551506 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.551413    2562 scope.go:117] "RemoveContainer" containerID="1a1d1a0f8ef94520bda9310bd172742ffccf798b3937086da000db2ff6bb63c1"
Sep 11 17:41:40.561312 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.561292    2562 scope.go:117] "RemoveContainer" containerID="81b6fa74b628ddea2f75a92623ce2bb97c698b6ae130ddbfed68ffdd92036003"
Sep 11 17:41:40.569713 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.569687    2562 scope.go:117] "RemoveContainer" containerID="1a1d1a0f8ef94520bda9310bd172742ffccf798b3937086da000db2ff6bb63c1"
Sep 11 17:41:40.570047 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:41:40.570027    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a1d1a0f8ef94520bda9310bd172742ffccf798b3937086da000db2ff6bb63c1\": container with ID starting with 1a1d1a0f8ef94520bda9310bd172742ffccf798b3937086da000db2ff6bb63c1 not found: ID does not exist" containerID="1a1d1a0f8ef94520bda9310bd172742ffccf798b3937086da000db2ff6bb63c1"
Sep 11 17:41:40.570120 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.570055    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a1d1a0f8ef94520bda9310bd172742ffccf798b3937086da000db2ff6bb63c1"} err="failed to get container status \"1a1d1a0f8ef94520bda9310bd172742ffccf798b3937086da000db2ff6bb63c1\": rpc error: code = NotFound desc = could not find container \"1a1d1a0f8ef94520bda9310bd172742ffccf798b3937086da000db2ff6bb63c1\": container with ID starting with 1a1d1a0f8ef94520bda9310bd172742ffccf798b3937086da000db2ff6bb63c1 not found: ID does not exist"
Sep 11 17:41:40.570120 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.570074    2562 scope.go:117] "RemoveContainer" containerID="81b6fa74b628ddea2f75a92623ce2bb97c698b6ae130ddbfed68ffdd92036003"
Sep 11 17:41:40.570340 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:41:40.570322    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81b6fa74b628ddea2f75a92623ce2bb97c698b6ae130ddbfed68ffdd92036003\": container with ID starting with 81b6fa74b628ddea2f75a92623ce2bb97c698b6ae130ddbfed68ffdd92036003 not found: ID does not exist" containerID="81b6fa74b628ddea2f75a92623ce2bb97c698b6ae130ddbfed68ffdd92036003"
Sep 11 17:41:40.570385 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.570347    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81b6fa74b628ddea2f75a92623ce2bb97c698b6ae130ddbfed68ffdd92036003"} err="failed to get container status \"81b6fa74b628ddea2f75a92623ce2bb97c698b6ae130ddbfed68ffdd92036003\": rpc error: code = NotFound desc = could not find container \"81b6fa74b628ddea2f75a92623ce2bb97c698b6ae130ddbfed68ffdd92036003\": container with ID starting with 81b6fa74b628ddea2f75a92623ce2bb97c698b6ae130ddbfed68ffdd92036003 not found: ID does not exist"
Sep 11 17:41:40.575262 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.575239    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"]
Sep 11 17:41:40.579435 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:40.579414    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6709f1a7-openshift-default-599dcf555b-phzql"]
Sep 11 17:41:42.126271 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:42.126234    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5eab92b-01a6-4f8e-a118-04d2db7fb047" path="/var/lib/kubelet/pods/e5eab92b-01a6-4f8e-a118-04d2db7fb047/volumes"
Sep 11 17:41:58.310653 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:58.310611    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-6fc9e6/auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5"]
Sep 11 17:41:58.311292 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:58.311269    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67502d7f-4095-4544-8561-231e3b23ceac" containerName="payload-pre-processing"
Sep 11 17:41:58.311377 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:58.311296    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="67502d7f-4095-4544-8561-231e3b23ceac" containerName="payload-pre-processing"
Sep 11 17:41:58.311377 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:58.311339    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5eab92b-01a6-4f8e-a118-04d2db7fb047" containerName="istio-proxy"
Sep 11 17:41:58.311377 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:58.311349    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5eab92b-01a6-4f8e-a118-04d2db7fb047" containerName="istio-proxy"
Sep 11 17:41:58.311377 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:58.311365    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5eab92b-01a6-4f8e-a118-04d2db7fb047" containerName="istio-proxy"
Sep 11 17:41:58.311377 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:58.311374    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5eab92b-01a6-4f8e-a118-04d2db7fb047" containerName="istio-proxy"
Sep 11 17:41:58.311538 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:58.311473    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5eab92b-01a6-4f8e-a118-04d2db7fb047" containerName="istio-proxy"
Sep 11 17:41:58.311538 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:58.311485    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="67502d7f-4095-4544-8561-231e3b23ceac" containerName="payload-pre-processing"
Sep 11 17:41:58.317072 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:58.315651    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5"
Sep 11 17:41:58.320165 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:58.320142    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6fc9e6\"/\"kube-root-ca.crt\""
Sep 11 17:41:58.320623 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:58.320602    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6fc9e6\"/\"auth-test-model-6fc9e6-kserve-self-signed-certs\""
Sep 11 17:41:58.321047 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:58.321029    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6fc9e6\"/\"default-dockercfg-kgvr7\""
Sep 11 17:41:58.321434 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:58.321414    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6fc9e6\"/\"openshift-service-ca.crt\""
Sep 11 17:41:58.324312 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:58.324279    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6fc9e6/auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5"]
Sep 11 17:41:58.386570 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:58.386529    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/671e43ce-84d2-4f03-b15c-dc98bec4d898-tls-certs\") pod \"auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5\" (UID: \"671e43ce-84d2-4f03-b15c-dc98bec4d898\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5"
Sep 11 17:41:58.386753 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:58.386591    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/671e43ce-84d2-4f03-b15c-dc98bec4d898-tmp-dir\") pod \"auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5\" (UID: \"671e43ce-84d2-4f03-b15c-dc98bec4d898\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5"
Sep 11 17:41:58.386753 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:58.386655    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/671e43ce-84d2-4f03-b15c-dc98bec4d898-dshm\") pod \"auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5\" (UID: \"671e43ce-84d2-4f03-b15c-dc98bec4d898\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5"
Sep 11 17:41:58.386753 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:58.386714    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7f92\" (UniqueName: \"kubernetes.io/projected/671e43ce-84d2-4f03-b15c-dc98bec4d898-kube-api-access-l7f92\") pod \"auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5\" (UID: \"671e43ce-84d2-4f03-b15c-dc98bec4d898\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5"
Sep 11 17:41:58.386753 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:58.386740    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/671e43ce-84d2-4f03-b15c-dc98bec4d898-model-cache\") pod \"auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5\" (UID: \"671e43ce-84d2-4f03-b15c-dc98bec4d898\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5"
Sep 11 17:41:58.386901 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:58.386830    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/671e43ce-84d2-4f03-b15c-dc98bec4d898-home\") pod \"auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5\" (UID: \"671e43ce-84d2-4f03-b15c-dc98bec4d898\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5"
Sep 11 17:41:58.487657 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:58.487616    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/671e43ce-84d2-4f03-b15c-dc98bec4d898-model-cache\") pod \"auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5\" (UID: \"671e43ce-84d2-4f03-b15c-dc98bec4d898\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5"
Sep 11 17:41:58.487873 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:58.487689    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/671e43ce-84d2-4f03-b15c-dc98bec4d898-home\") pod \"auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5\" (UID: \"671e43ce-84d2-4f03-b15c-dc98bec4d898\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5"
Sep 11 17:41:58.487873 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:58.487727    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/671e43ce-84d2-4f03-b15c-dc98bec4d898-tls-certs\") pod \"auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5\" (UID: \"671e43ce-84d2-4f03-b15c-dc98bec4d898\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5"
Sep 11 17:41:58.487873 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:58.487770    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/671e43ce-84d2-4f03-b15c-dc98bec4d898-tmp-dir\") pod \"auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5\" (UID: \"671e43ce-84d2-4f03-b15c-dc98bec4d898\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5"
Sep 11 17:41:58.487873 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:58.487795    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/671e43ce-84d2-4f03-b15c-dc98bec4d898-dshm\") pod \"auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5\" (UID: \"671e43ce-84d2-4f03-b15c-dc98bec4d898\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5"
Sep 11 17:41:58.487873 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:58.487848    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7f92\" (UniqueName: \"kubernetes.io/projected/671e43ce-84d2-4f03-b15c-dc98bec4d898-kube-api-access-l7f92\") pod \"auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5\" (UID: \"671e43ce-84d2-4f03-b15c-dc98bec4d898\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5"
Sep 11 17:41:58.488283 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:58.488148    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/671e43ce-84d2-4f03-b15c-dc98bec4d898-model-cache\") pod \"auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5\" (UID: \"671e43ce-84d2-4f03-b15c-dc98bec4d898\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5"
Sep 11 17:41:58.488283 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:58.488199    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/671e43ce-84d2-4f03-b15c-dc98bec4d898-home\") pod \"auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5\" (UID: \"671e43ce-84d2-4f03-b15c-dc98bec4d898\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5"
Sep 11 17:41:58.488283 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:58.488253    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/671e43ce-84d2-4f03-b15c-dc98bec4d898-tmp-dir\") pod \"auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5\" (UID: \"671e43ce-84d2-4f03-b15c-dc98bec4d898\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5"
Sep 11 17:41:58.490102 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:58.490080    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/671e43ce-84d2-4f03-b15c-dc98bec4d898-dshm\") pod \"auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5\" (UID: \"671e43ce-84d2-4f03-b15c-dc98bec4d898\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5"
Sep 11 17:41:58.490353 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:58.490332    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/671e43ce-84d2-4f03-b15c-dc98bec4d898-tls-certs\") pod \"auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5\" (UID: \"671e43ce-84d2-4f03-b15c-dc98bec4d898\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5"
Sep 11 17:41:58.497422 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:58.497398    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7f92\" (UniqueName: \"kubernetes.io/projected/671e43ce-84d2-4f03-b15c-dc98bec4d898-kube-api-access-l7f92\") pod \"auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5\" (UID: \"671e43ce-84d2-4f03-b15c-dc98bec4d898\") " pod="ai-tenant-e2e-shared-b-w3-6fc9e6/auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5"
Sep 11 17:41:58.630771 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:58.630724    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5"
Sep 11 17:41:58.904649 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:58.904622    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6fc9e6/auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5"]
Sep 11 17:41:58.906578 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:41:58.906549    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod671e43ce_84d2_4f03_b15c_dc98bec4d898.slice/crio-44cb2b2fa64d07882bf9f3838aa44a79d9487e2ad4d8883bba6edd94b466e96e WatchSource:0}: Error finding container 44cb2b2fa64d07882bf9f3838aa44a79d9487e2ad4d8883bba6edd94b466e96e: Status 404 returned error can't find the container with id 44cb2b2fa64d07882bf9f3838aa44a79d9487e2ad4d8883bba6edd94b466e96e
Sep 11 17:41:59.649527 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:59.649487    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5" event={"ID":"671e43ce-84d2-4f03-b15c-dc98bec4d898","Type":"ContainerStarted","Data":"1b88f61bf26d44c43308ac5d3df3968065b2cff082ec02e308a73113811d6432"}
Sep 11 17:41:59.649527 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:59.649531    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5" event={"ID":"671e43ce-84d2-4f03-b15c-dc98bec4d898","Type":"ContainerStarted","Data":"44cb2b2fa64d07882bf9f3838aa44a79d9487e2ad4d8883bba6edd94b466e96e"}
Sep 11 17:41:59.666871 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:41:59.666816    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5" podStartSLOduration=1.666799216 podStartE2EDuration="1.666799216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:41:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:41:59.665829076 +0000 UTC m=+1400.059746531" watchObservedRunningTime="2026-09-11 17:41:59.666799216 +0000 UTC m=+1400.060716670"
Sep 11 17:42:08.631553 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:42:08.631508    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5"
Sep 11 17:42:08.631986 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:42:08.631570    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5"
Sep 11 17:42:08.644499 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:42:08.644470    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5"
Sep 11 17:42:08.704931 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:42:08.704900    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5"
Sep 11 17:43:04.714882 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:04.714848    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"]
Sep 11 17:43:04.715423 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:04.715362    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5eab92b-01a6-4f8e-a118-04d2db7fb047" containerName="istio-proxy"
Sep 11 17:43:04.718210 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:04.718186    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"
Sep 11 17:43:04.723341 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:04.723321    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-83a2bc-openshift-default-dockercfg-gqhxf\""
Sep 11 17:43:04.739119 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:04.739095    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"]
Sep 11 17:43:04.863418 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:04.863375    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/653eff13-fedd-469b-b0ac-e9d51e6da73b-istio-envoy\") pod \"e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb\" (UID: \"653eff13-fedd-469b-b0ac-e9d51e6da73b\") " pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"
Sep 11 17:43:04.863418 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:04.863415    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/653eff13-fedd-469b-b0ac-e9d51e6da73b-istiod-ca-cert\") pod \"e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb\" (UID: \"653eff13-fedd-469b-b0ac-e9d51e6da73b\") " pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"
Sep 11 17:43:04.863683 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:04.863507    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/653eff13-fedd-469b-b0ac-e9d51e6da73b-istio-token\") pod \"e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb\" (UID: \"653eff13-fedd-469b-b0ac-e9d51e6da73b\") " pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"
Sep 11 17:43:04.863683 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:04.863536    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/653eff13-fedd-469b-b0ac-e9d51e6da73b-credential-socket\") pod \"e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb\" (UID: \"653eff13-fedd-469b-b0ac-e9d51e6da73b\") " pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"
Sep 11 17:43:04.863683 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:04.863582    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/653eff13-fedd-469b-b0ac-e9d51e6da73b-istio-podinfo\") pod \"e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb\" (UID: \"653eff13-fedd-469b-b0ac-e9d51e6da73b\") " pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"
Sep 11 17:43:04.863683 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:04.863648    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/653eff13-fedd-469b-b0ac-e9d51e6da73b-istio-data\") pod \"e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb\" (UID: \"653eff13-fedd-469b-b0ac-e9d51e6da73b\") " pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"
Sep 11 17:43:04.863683 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:04.863664    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk2xm\" (UniqueName: \"kubernetes.io/projected/653eff13-fedd-469b-b0ac-e9d51e6da73b-kube-api-access-sk2xm\") pod \"e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb\" (UID: \"653eff13-fedd-469b-b0ac-e9d51e6da73b\") " pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"
Sep 11 17:43:04.863683 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:04.863682    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/653eff13-fedd-469b-b0ac-e9d51e6da73b-workload-socket\") pod \"e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb\" (UID: \"653eff13-fedd-469b-b0ac-e9d51e6da73b\") " pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"
Sep 11 17:43:04.863907 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:04.863701    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/653eff13-fedd-469b-b0ac-e9d51e6da73b-workload-certs\") pod \"e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb\" (UID: \"653eff13-fedd-469b-b0ac-e9d51e6da73b\") " pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"
Sep 11 17:43:04.964487 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:04.964454    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/653eff13-fedd-469b-b0ac-e9d51e6da73b-istio-data\") pod \"e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb\" (UID: \"653eff13-fedd-469b-b0ac-e9d51e6da73b\") " pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"
Sep 11 17:43:04.964487 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:04.964492    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk2xm\" (UniqueName: \"kubernetes.io/projected/653eff13-fedd-469b-b0ac-e9d51e6da73b-kube-api-access-sk2xm\") pod \"e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb\" (UID: \"653eff13-fedd-469b-b0ac-e9d51e6da73b\") " pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"
Sep 11 17:43:04.964714 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:04.964582    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/653eff13-fedd-469b-b0ac-e9d51e6da73b-workload-socket\") pod \"e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb\" (UID: \"653eff13-fedd-469b-b0ac-e9d51e6da73b\") " pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"
Sep 11 17:43:04.964714 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:04.964613    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/653eff13-fedd-469b-b0ac-e9d51e6da73b-workload-certs\") pod \"e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb\" (UID: \"653eff13-fedd-469b-b0ac-e9d51e6da73b\") " pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"
Sep 11 17:43:04.964714 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:04.964685    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/653eff13-fedd-469b-b0ac-e9d51e6da73b-istio-envoy\") pod \"e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb\" (UID: \"653eff13-fedd-469b-b0ac-e9d51e6da73b\") " pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"
Sep 11 17:43:04.964714 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:04.964707    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/653eff13-fedd-469b-b0ac-e9d51e6da73b-istiod-ca-cert\") pod \"e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb\" (UID: \"653eff13-fedd-469b-b0ac-e9d51e6da73b\") " pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"
Sep 11 17:43:04.964943 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:04.964796    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/653eff13-fedd-469b-b0ac-e9d51e6da73b-istio-token\") pod \"e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb\" (UID: \"653eff13-fedd-469b-b0ac-e9d51e6da73b\") " pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"
Sep 11 17:43:04.964943 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:04.964852    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/653eff13-fedd-469b-b0ac-e9d51e6da73b-credential-socket\") pod \"e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb\" (UID: \"653eff13-fedd-469b-b0ac-e9d51e6da73b\") " pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"
Sep 11 17:43:04.964943 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:04.964887    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/653eff13-fedd-469b-b0ac-e9d51e6da73b-istio-podinfo\") pod \"e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb\" (UID: \"653eff13-fedd-469b-b0ac-e9d51e6da73b\") " pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"
Sep 11 17:43:04.964943 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:04.964897    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/653eff13-fedd-469b-b0ac-e9d51e6da73b-istio-data\") pod \"e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb\" (UID: \"653eff13-fedd-469b-b0ac-e9d51e6da73b\") " pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"
Sep 11 17:43:04.965188 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:04.964970    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/653eff13-fedd-469b-b0ac-e9d51e6da73b-workload-certs\") pod \"e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb\" (UID: \"653eff13-fedd-469b-b0ac-e9d51e6da73b\") " pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"
Sep 11 17:43:04.965188 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:04.965104    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/653eff13-fedd-469b-b0ac-e9d51e6da73b-workload-socket\") pod \"e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb\" (UID: \"653eff13-fedd-469b-b0ac-e9d51e6da73b\") " pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"
Sep 11 17:43:04.965270 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:04.965219    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/653eff13-fedd-469b-b0ac-e9d51e6da73b-credential-socket\") pod \"e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb\" (UID: \"653eff13-fedd-469b-b0ac-e9d51e6da73b\") " pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"
Sep 11 17:43:04.965649 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:04.965629    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/653eff13-fedd-469b-b0ac-e9d51e6da73b-istiod-ca-cert\") pod \"e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb\" (UID: \"653eff13-fedd-469b-b0ac-e9d51e6da73b\") " pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"
Sep 11 17:43:04.967099 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:04.967076    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/653eff13-fedd-469b-b0ac-e9d51e6da73b-istio-envoy\") pod \"e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb\" (UID: \"653eff13-fedd-469b-b0ac-e9d51e6da73b\") " pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"
Sep 11 17:43:04.967300 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:04.967283    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/653eff13-fedd-469b-b0ac-e9d51e6da73b-istio-podinfo\") pod \"e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb\" (UID: \"653eff13-fedd-469b-b0ac-e9d51e6da73b\") " pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"
Sep 11 17:43:05.010074 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:05.010037    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/653eff13-fedd-469b-b0ac-e9d51e6da73b-istio-token\") pod \"e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb\" (UID: \"653eff13-fedd-469b-b0ac-e9d51e6da73b\") " pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"
Sep 11 17:43:05.010228 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:05.010121    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk2xm\" (UniqueName: \"kubernetes.io/projected/653eff13-fedd-469b-b0ac-e9d51e6da73b-kube-api-access-sk2xm\") pod \"e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb\" (UID: \"653eff13-fedd-469b-b0ac-e9d51e6da73b\") " pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"
Sep 11 17:43:05.028980 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:05.028941    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"
Sep 11 17:43:05.313396 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:05.313371    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"]
Sep 11 17:43:05.314606 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:43:05.314579    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod653eff13_fedd_469b_b0ac_e9d51e6da73b.slice/crio-582074be771a280e5dd47a912cc2f3e58c50bcc6e1526e5ec08386918752d66a WatchSource:0}: Error finding container 582074be771a280e5dd47a912cc2f3e58c50bcc6e1526e5ec08386918752d66a: Status 404 returned error can't find the container with id 582074be771a280e5dd47a912cc2f3e58c50bcc6e1526e5ec08386918752d66a
Sep 11 17:43:05.317113 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:05.317083    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:43:05.317195 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:05.317149    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:43:05.317195 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:05.317179    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:43:05.986616 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:05.986580    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb" event={"ID":"653eff13-fedd-469b-b0ac-e9d51e6da73b","Type":"ContainerStarted","Data":"1db6ab532ec5d2ead0c7912c77352c6712b5770fd8002936b22fcc63dca89503"}
Sep 11 17:43:05.987053 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:05.986644    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb" event={"ID":"653eff13-fedd-469b-b0ac-e9d51e6da73b","Type":"ContainerStarted","Data":"582074be771a280e5dd47a912cc2f3e58c50bcc6e1526e5ec08386918752d66a"}
Sep 11 17:43:06.029460 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:06.029431    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"
Sep 11 17:43:06.034774 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:06.034743    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"
Sep 11 17:43:06.066787 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:06.066730    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb" podStartSLOduration=2.066712646 podStartE2EDuration="2.066712646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:43:06.024370594 +0000 UTC m=+1466.418288051" watchObservedRunningTime="2026-09-11 17:43:06.066712646 +0000 UTC m=+1466.460630101"
Sep 11 17:43:06.992796 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:06.992768    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"
Sep 11 17:43:06.993717 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:06.993691    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"
Sep 11 17:43:12.461101 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:12.461061    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-83a2bc-5484b775f-gfc2l"]
Sep 11 17:43:12.464440 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:12.464414    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-83a2bc-5484b775f-gfc2l"
Sep 11 17:43:12.468353 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:12.468315    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-83a2bc-dockercfg-8k2dz\""
Sep 11 17:43:12.468616 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:12.468592    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-83a2bc\""
Sep 11 17:43:12.476115 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:12.476088    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-83a2bc-5484b775f-gfc2l"]
Sep 11 17:43:12.551763 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:12.551726    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j9jp\" (UniqueName: \"kubernetes.io/projected/34712b21-424b-495f-a78a-e42d802483de-kube-api-access-8j9jp\") pod \"payload-processing-e2e-api-b-83a2bc-5484b775f-gfc2l\" (UID: \"34712b21-424b-495f-a78a-e42d802483de\") " pod="openshift-ingress/payload-processing-e2e-api-b-83a2bc-5484b775f-gfc2l"
Sep 11 17:43:12.551763 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:12.551765    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34712b21-424b-495f-a78a-e42d802483de-plugins-config-volume\") pod \"payload-processing-e2e-api-b-83a2bc-5484b775f-gfc2l\" (UID: \"34712b21-424b-495f-a78a-e42d802483de\") " pod="openshift-ingress/payload-processing-e2e-api-b-83a2bc-5484b775f-gfc2l"
Sep 11 17:43:12.652928 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:12.652892    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j9jp\" (UniqueName: \"kubernetes.io/projected/34712b21-424b-495f-a78a-e42d802483de-kube-api-access-8j9jp\") pod \"payload-processing-e2e-api-b-83a2bc-5484b775f-gfc2l\" (UID: \"34712b21-424b-495f-a78a-e42d802483de\") " pod="openshift-ingress/payload-processing-e2e-api-b-83a2bc-5484b775f-gfc2l"
Sep 11 17:43:12.652928 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:12.652933    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34712b21-424b-495f-a78a-e42d802483de-plugins-config-volume\") pod \"payload-processing-e2e-api-b-83a2bc-5484b775f-gfc2l\" (UID: \"34712b21-424b-495f-a78a-e42d802483de\") " pod="openshift-ingress/payload-processing-e2e-api-b-83a2bc-5484b775f-gfc2l"
Sep 11 17:43:12.653656 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:12.653638    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34712b21-424b-495f-a78a-e42d802483de-plugins-config-volume\") pod \"payload-processing-e2e-api-b-83a2bc-5484b775f-gfc2l\" (UID: \"34712b21-424b-495f-a78a-e42d802483de\") " pod="openshift-ingress/payload-processing-e2e-api-b-83a2bc-5484b775f-gfc2l"
Sep 11 17:43:12.660708 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:12.660684    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j9jp\" (UniqueName: \"kubernetes.io/projected/34712b21-424b-495f-a78a-e42d802483de-kube-api-access-8j9jp\") pod \"payload-processing-e2e-api-b-83a2bc-5484b775f-gfc2l\" (UID: \"34712b21-424b-495f-a78a-e42d802483de\") " pod="openshift-ingress/payload-processing-e2e-api-b-83a2bc-5484b775f-gfc2l"
Sep 11 17:43:12.778379 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:12.778293    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-83a2bc-5484b775f-gfc2l"
Sep 11 17:43:13.051211 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:13.051184    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-83a2bc-5484b775f-gfc2l"]
Sep 11 17:43:13.053257 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:43:13.053214    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34712b21_424b_495f_a78a_e42d802483de.slice/crio-2ac9c6c8291a099a8d90497d96e7abb2a5b8a450fe0a2dd8e988de974f740257 WatchSource:0}: Error finding container 2ac9c6c8291a099a8d90497d96e7abb2a5b8a450fe0a2dd8e988de974f740257: Status 404 returned error can't find the container with id 2ac9c6c8291a099a8d90497d96e7abb2a5b8a450fe0a2dd8e988de974f740257
Sep 11 17:43:14.049381 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:14.049337    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-83a2bc-5484b775f-gfc2l" event={"ID":"34712b21-424b-495f-a78a-e42d802483de","Type":"ContainerStarted","Data":"68303079b24d7f136e101e8ea75971598254662b6e10ead5fdecf8cf3c258ca7"}
Sep 11 17:43:14.049824 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:14.049417    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-83a2bc-5484b775f-gfc2l" event={"ID":"34712b21-424b-495f-a78a-e42d802483de","Type":"ContainerStarted","Data":"2ac9c6c8291a099a8d90497d96e7abb2a5b8a450fe0a2dd8e988de974f740257"}
Sep 11 17:43:14.049824 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:14.049486    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-83a2bc-5484b775f-gfc2l"
Sep 11 17:43:14.067426 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:14.067369    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-83a2bc-5484b775f-gfc2l" podStartSLOduration=2.067351556 podStartE2EDuration="2.067351556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:43:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:43:14.06604074 +0000 UTC m=+1474.459958195" watchObservedRunningTime="2026-09-11 17:43:14.067351556 +0000 UTC m=+1474.461269010"
Sep 11 17:43:25.058910 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:25.058877    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-83a2bc-5484b775f-gfc2l"
Sep 11 17:43:43.212610 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:43.212576    2562 generic.go:356] "Generic (PLEG): container finished" podID="653eff13-fedd-469b-b0ac-e9d51e6da73b" containerID="1db6ab532ec5d2ead0c7912c77352c6712b5770fd8002936b22fcc63dca89503" exitCode=0
Sep 11 17:43:43.213045 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:43.212647    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb" event={"ID":"653eff13-fedd-469b-b0ac-e9d51e6da73b","Type":"ContainerDied","Data":"1db6ab532ec5d2ead0c7912c77352c6712b5770fd8002936b22fcc63dca89503"}
Sep 11 17:43:43.213122 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:43.213105    2562 scope.go:117] "RemoveContainer" containerID="1db6ab532ec5d2ead0c7912c77352c6712b5770fd8002936b22fcc63dca89503"
Sep 11 17:43:43.214380 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:43.214355    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:43:43.214443 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:43.214422    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:43:43.214479 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:43.214454    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:43:44.224122 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:44.224085    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb" event={"ID":"653eff13-fedd-469b-b0ac-e9d51e6da73b","Type":"ContainerStarted","Data":"72890d898409a05a6f538c04f449a7d8fdb5b5e1a7ab24bbee9d2dd1f849355d"}
Sep 11 17:43:45.029452 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:45.029415    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"
Sep 11 17:43:45.029690 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:45.029665    2562 patch_prober.go:28] interesting pod/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 11 17:43:45.029758 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:45.029732    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb" podUID="653eff13-fedd-469b-b0ac-e9d51e6da73b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 11 17:43:46.030044 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:46.030009    2562 patch_prober.go:28] interesting pod/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 11 17:43:46.030431 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:46.030077    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb" podUID="653eff13-fedd-469b-b0ac-e9d51e6da73b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 11 17:43:47.029962 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:47.029917    2562 patch_prober.go:28] interesting pod/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 11 17:43:47.030181 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:47.029981    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb" podUID="653eff13-fedd-469b-b0ac-e9d51e6da73b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 11 17:43:48.029622 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:48.029574    2562 patch_prober.go:28] interesting pod/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 11 17:43:48.029852 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:48.029636    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb" podUID="653eff13-fedd-469b-b0ac-e9d51e6da73b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 11 17:43:49.029574 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:49.029538    2562 patch_prober.go:28] interesting pod/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 11 17:43:49.030020 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:49.029603    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb" podUID="653eff13-fedd-469b-b0ac-e9d51e6da73b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 11 17:43:50.033988 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:50.033956    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"
Sep 11 17:43:50.034503 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:50.034478    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"
Sep 11 17:43:50.035106 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:43:50.035066    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"
Sep 11 17:44:20.778409 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:20.778370    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-83a2bc-5484b775f-gfc2l"]
Sep 11 17:44:20.778917 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:20.778633    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-83a2bc-5484b775f-gfc2l" podUID="34712b21-424b-495f-a78a-e42d802483de" containerName="payload-processing" containerID="cri-o://68303079b24d7f136e101e8ea75971598254662b6e10ead5fdecf8cf3c258ca7" gracePeriod=30
Sep 11 17:44:21.068761 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:21.068732    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-83a2bc-5484b775f-gfc2l"
Sep 11 17:44:21.177858 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:21.177816    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8j9jp\" (UniqueName: \"kubernetes.io/projected/34712b21-424b-495f-a78a-e42d802483de-kube-api-access-8j9jp\") pod \"34712b21-424b-495f-a78a-e42d802483de\" (UID: \"34712b21-424b-495f-a78a-e42d802483de\") "
Sep 11 17:44:21.178084 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:21.177922    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34712b21-424b-495f-a78a-e42d802483de-plugins-config-volume\") pod \"34712b21-424b-495f-a78a-e42d802483de\" (UID: \"34712b21-424b-495f-a78a-e42d802483de\") "
Sep 11 17:44:21.178339 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:21.178295    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34712b21-424b-495f-a78a-e42d802483de-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "34712b21-424b-495f-a78a-e42d802483de" (UID: "34712b21-424b-495f-a78a-e42d802483de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:44:21.180145 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:21.180114    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34712b21-424b-495f-a78a-e42d802483de-kube-api-access-8j9jp" (OuterVolumeSpecName: "kube-api-access-8j9jp") pod "34712b21-424b-495f-a78a-e42d802483de" (UID: "34712b21-424b-495f-a78a-e42d802483de"). InnerVolumeSpecName "kube-api-access-8j9jp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:44:21.279368 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:21.279336    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j9jp\" (UniqueName: \"kubernetes.io/projected/34712b21-424b-495f-a78a-e42d802483de-kube-api-access-8j9jp\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:44:21.279368 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:21.279366    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34712b21-424b-495f-a78a-e42d802483de-plugins-config-volume\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:44:21.432403 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:21.432372    2562 generic.go:356] "Generic (PLEG): container finished" podID="34712b21-424b-495f-a78a-e42d802483de" containerID="68303079b24d7f136e101e8ea75971598254662b6e10ead5fdecf8cf3c258ca7" exitCode=0
Sep 11 17:44:21.432582 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:21.432420    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-83a2bc-5484b775f-gfc2l"
Sep 11 17:44:21.432582 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:21.432467    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-83a2bc-5484b775f-gfc2l" event={"ID":"34712b21-424b-495f-a78a-e42d802483de","Type":"ContainerDied","Data":"68303079b24d7f136e101e8ea75971598254662b6e10ead5fdecf8cf3c258ca7"}
Sep 11 17:44:21.432582 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:21.432519    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-83a2bc-5484b775f-gfc2l" event={"ID":"34712b21-424b-495f-a78a-e42d802483de","Type":"ContainerDied","Data":"2ac9c6c8291a099a8d90497d96e7abb2a5b8a450fe0a2dd8e988de974f740257"}
Sep 11 17:44:21.432582 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:21.432546    2562 scope.go:117] "RemoveContainer" containerID="68303079b24d7f136e101e8ea75971598254662b6e10ead5fdecf8cf3c258ca7"
Sep 11 17:44:21.442677 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:21.442660    2562 scope.go:117] "RemoveContainer" containerID="68303079b24d7f136e101e8ea75971598254662b6e10ead5fdecf8cf3c258ca7"
Sep 11 17:44:21.442936 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:44:21.442917    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68303079b24d7f136e101e8ea75971598254662b6e10ead5fdecf8cf3c258ca7\": container with ID starting with 68303079b24d7f136e101e8ea75971598254662b6e10ead5fdecf8cf3c258ca7 not found: ID does not exist" containerID="68303079b24d7f136e101e8ea75971598254662b6e10ead5fdecf8cf3c258ca7"
Sep 11 17:44:21.443006 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:21.442945    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68303079b24d7f136e101e8ea75971598254662b6e10ead5fdecf8cf3c258ca7"} err="failed to get container status \"68303079b24d7f136e101e8ea75971598254662b6e10ead5fdecf8cf3c258ca7\": rpc error: code = NotFound desc = could not find container \"68303079b24d7f136e101e8ea75971598254662b6e10ead5fdecf8cf3c258ca7\": container with ID starting with 68303079b24d7f136e101e8ea75971598254662b6e10ead5fdecf8cf3c258ca7 not found: ID does not exist"
Sep 11 17:44:21.456825 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:21.456797    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-83a2bc-5484b775f-gfc2l"]
Sep 11 17:44:21.463793 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:21.463766    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-83a2bc-5484b775f-gfc2l"]
Sep 11 17:44:22.130018 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:22.129960    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34712b21-424b-495f-a78a-e42d802483de" path="/var/lib/kubelet/pods/34712b21-424b-495f-a78a-e42d802483de/volumes"
Sep 11 17:44:29.401388 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:29.401356    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"]
Sep 11 17:44:29.401810 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:29.401621    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb" podUID="653eff13-fedd-469b-b0ac-e9d51e6da73b" containerName="istio-proxy" containerID="cri-o://72890d898409a05a6f538c04f449a7d8fdb5b5e1a7ab24bbee9d2dd1f849355d" gracePeriod=30
Sep 11 17:44:34.666135 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:34.666066    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"
Sep 11 17:44:34.821118 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:34.821080    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/653eff13-fedd-469b-b0ac-e9d51e6da73b-workload-socket\") pod \"653eff13-fedd-469b-b0ac-e9d51e6da73b\" (UID: \"653eff13-fedd-469b-b0ac-e9d51e6da73b\") "
Sep 11 17:44:34.821316 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:34.821128    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/653eff13-fedd-469b-b0ac-e9d51e6da73b-istio-data\") pod \"653eff13-fedd-469b-b0ac-e9d51e6da73b\" (UID: \"653eff13-fedd-469b-b0ac-e9d51e6da73b\") "
Sep 11 17:44:34.821316 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:34.821182    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/653eff13-fedd-469b-b0ac-e9d51e6da73b-credential-socket\") pod \"653eff13-fedd-469b-b0ac-e9d51e6da73b\" (UID: \"653eff13-fedd-469b-b0ac-e9d51e6da73b\") "
Sep 11 17:44:34.821316 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:34.821207    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/653eff13-fedd-469b-b0ac-e9d51e6da73b-workload-certs\") pod \"653eff13-fedd-469b-b0ac-e9d51e6da73b\" (UID: \"653eff13-fedd-469b-b0ac-e9d51e6da73b\") "
Sep 11 17:44:34.821316 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:34.821239    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sk2xm\" (UniqueName: \"kubernetes.io/projected/653eff13-fedd-469b-b0ac-e9d51e6da73b-kube-api-access-sk2xm\") pod \"653eff13-fedd-469b-b0ac-e9d51e6da73b\" (UID: \"653eff13-fedd-469b-b0ac-e9d51e6da73b\") "
Sep 11 17:44:34.821316 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:34.821290    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/653eff13-fedd-469b-b0ac-e9d51e6da73b-istio-podinfo\") pod \"653eff13-fedd-469b-b0ac-e9d51e6da73b\" (UID: \"653eff13-fedd-469b-b0ac-e9d51e6da73b\") "
Sep 11 17:44:34.821316 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:34.821313    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/653eff13-fedd-469b-b0ac-e9d51e6da73b-istio-token\") pod \"653eff13-fedd-469b-b0ac-e9d51e6da73b\" (UID: \"653eff13-fedd-469b-b0ac-e9d51e6da73b\") "
Sep 11 17:44:34.821624 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:34.821405    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/653eff13-fedd-469b-b0ac-e9d51e6da73b-istiod-ca-cert\") pod \"653eff13-fedd-469b-b0ac-e9d51e6da73b\" (UID: \"653eff13-fedd-469b-b0ac-e9d51e6da73b\") "
Sep 11 17:44:34.821624 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:34.821436    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/653eff13-fedd-469b-b0ac-e9d51e6da73b-istio-envoy\") pod \"653eff13-fedd-469b-b0ac-e9d51e6da73b\" (UID: \"653eff13-fedd-469b-b0ac-e9d51e6da73b\") "
Sep 11 17:44:34.821624 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:34.821438    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/653eff13-fedd-469b-b0ac-e9d51e6da73b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "653eff13-fedd-469b-b0ac-e9d51e6da73b" (UID: "653eff13-fedd-469b-b0ac-e9d51e6da73b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:44:34.821624 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:34.821529    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/653eff13-fedd-469b-b0ac-e9d51e6da73b-istio-data" (OuterVolumeSpecName: "istio-data") pod "653eff13-fedd-469b-b0ac-e9d51e6da73b" (UID: "653eff13-fedd-469b-b0ac-e9d51e6da73b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:44:34.821837 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:34.821819    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/653eff13-fedd-469b-b0ac-e9d51e6da73b-workload-socket\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:44:34.821894 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:34.821837    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/653eff13-fedd-469b-b0ac-e9d51e6da73b-istio-data\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:44:34.821894 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:34.821868    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/653eff13-fedd-469b-b0ac-e9d51e6da73b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "653eff13-fedd-469b-b0ac-e9d51e6da73b" (UID: "653eff13-fedd-469b-b0ac-e9d51e6da73b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:44:34.822020 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:34.821886    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/653eff13-fedd-469b-b0ac-e9d51e6da73b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "653eff13-fedd-469b-b0ac-e9d51e6da73b" (UID: "653eff13-fedd-469b-b0ac-e9d51e6da73b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:44:34.822020 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:34.821935    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/653eff13-fedd-469b-b0ac-e9d51e6da73b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "653eff13-fedd-469b-b0ac-e9d51e6da73b" (UID: "653eff13-fedd-469b-b0ac-e9d51e6da73b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:44:34.823666 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:34.823637    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/653eff13-fedd-469b-b0ac-e9d51e6da73b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "653eff13-fedd-469b-b0ac-e9d51e6da73b" (UID: "653eff13-fedd-469b-b0ac-e9d51e6da73b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 17:44:34.823780 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:34.823675    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/653eff13-fedd-469b-b0ac-e9d51e6da73b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "653eff13-fedd-469b-b0ac-e9d51e6da73b" (UID: "653eff13-fedd-469b-b0ac-e9d51e6da73b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:44:34.823925 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:34.823905    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/653eff13-fedd-469b-b0ac-e9d51e6da73b-istio-token" (OuterVolumeSpecName: "istio-token") pod "653eff13-fedd-469b-b0ac-e9d51e6da73b" (UID: "653eff13-fedd-469b-b0ac-e9d51e6da73b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:44:34.824014 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:34.823967    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/653eff13-fedd-469b-b0ac-e9d51e6da73b-kube-api-access-sk2xm" (OuterVolumeSpecName: "kube-api-access-sk2xm") pod "653eff13-fedd-469b-b0ac-e9d51e6da73b" (UID: "653eff13-fedd-469b-b0ac-e9d51e6da73b"). InnerVolumeSpecName "kube-api-access-sk2xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:44:34.923381 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:34.923292    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/653eff13-fedd-469b-b0ac-e9d51e6da73b-credential-socket\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:44:34.923381 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:34.923324    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/653eff13-fedd-469b-b0ac-e9d51e6da73b-workload-certs\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:44:34.923381 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:34.923335    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk2xm\" (UniqueName: \"kubernetes.io/projected/653eff13-fedd-469b-b0ac-e9d51e6da73b-kube-api-access-sk2xm\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:44:34.923381 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:34.923346    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/653eff13-fedd-469b-b0ac-e9d51e6da73b-istio-podinfo\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:44:34.923381 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:34.923355    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/653eff13-fedd-469b-b0ac-e9d51e6da73b-istio-token\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:44:34.923381 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:34.923363    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/653eff13-fedd-469b-b0ac-e9d51e6da73b-istiod-ca-cert\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:44:34.923381 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:34.923371    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/653eff13-fedd-469b-b0ac-e9d51e6da73b-istio-envoy\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:44:35.511728 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:35.511692    2562 generic.go:356] "Generic (PLEG): container finished" podID="653eff13-fedd-469b-b0ac-e9d51e6da73b" containerID="72890d898409a05a6f538c04f449a7d8fdb5b5e1a7ab24bbee9d2dd1f849355d" exitCode=0
Sep 11 17:44:35.511910 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:35.511741    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"
Sep 11 17:44:35.511910 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:35.511769    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb" event={"ID":"653eff13-fedd-469b-b0ac-e9d51e6da73b","Type":"ContainerDied","Data":"72890d898409a05a6f538c04f449a7d8fdb5b5e1a7ab24bbee9d2dd1f849355d"}
Sep 11 17:44:35.511910 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:35.511832    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb" event={"ID":"653eff13-fedd-469b-b0ac-e9d51e6da73b","Type":"ContainerDied","Data":"582074be771a280e5dd47a912cc2f3e58c50bcc6e1526e5ec08386918752d66a"}
Sep 11 17:44:35.511910 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:35.511863    2562 scope.go:117] "RemoveContainer" containerID="72890d898409a05a6f538c04f449a7d8fdb5b5e1a7ab24bbee9d2dd1f849355d"
Sep 11 17:44:35.522223 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:35.522053    2562 scope.go:117] "RemoveContainer" containerID="1db6ab532ec5d2ead0c7912c77352c6712b5770fd8002936b22fcc63dca89503"
Sep 11 17:44:35.530773 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:35.530745    2562 scope.go:117] "RemoveContainer" containerID="72890d898409a05a6f538c04f449a7d8fdb5b5e1a7ab24bbee9d2dd1f849355d"
Sep 11 17:44:35.531080 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:44:35.531058    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72890d898409a05a6f538c04f449a7d8fdb5b5e1a7ab24bbee9d2dd1f849355d\": container with ID starting with 72890d898409a05a6f538c04f449a7d8fdb5b5e1a7ab24bbee9d2dd1f849355d not found: ID does not exist" containerID="72890d898409a05a6f538c04f449a7d8fdb5b5e1a7ab24bbee9d2dd1f849355d"
Sep 11 17:44:35.531163 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:35.531093    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72890d898409a05a6f538c04f449a7d8fdb5b5e1a7ab24bbee9d2dd1f849355d"} err="failed to get container status \"72890d898409a05a6f538c04f449a7d8fdb5b5e1a7ab24bbee9d2dd1f849355d\": rpc error: code = NotFound desc = could not find container \"72890d898409a05a6f538c04f449a7d8fdb5b5e1a7ab24bbee9d2dd1f849355d\": container with ID starting with 72890d898409a05a6f538c04f449a7d8fdb5b5e1a7ab24bbee9d2dd1f849355d not found: ID does not exist"
Sep 11 17:44:35.531163 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:35.531116    2562 scope.go:117] "RemoveContainer" containerID="1db6ab532ec5d2ead0c7912c77352c6712b5770fd8002936b22fcc63dca89503"
Sep 11 17:44:35.531396 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:44:35.531380    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1db6ab532ec5d2ead0c7912c77352c6712b5770fd8002936b22fcc63dca89503\": container with ID starting with 1db6ab532ec5d2ead0c7912c77352c6712b5770fd8002936b22fcc63dca89503 not found: ID does not exist" containerID="1db6ab532ec5d2ead0c7912c77352c6712b5770fd8002936b22fcc63dca89503"
Sep 11 17:44:35.531451 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:35.531403    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1db6ab532ec5d2ead0c7912c77352c6712b5770fd8002936b22fcc63dca89503"} err="failed to get container status \"1db6ab532ec5d2ead0c7912c77352c6712b5770fd8002936b22fcc63dca89503\": rpc error: code = NotFound desc = could not find container \"1db6ab532ec5d2ead0c7912c77352c6712b5770fd8002936b22fcc63dca89503\": container with ID starting with 1db6ab532ec5d2ead0c7912c77352c6712b5770fd8002936b22fcc63dca89503 not found: ID does not exist"
Sep 11 17:44:35.536056 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:35.536030    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"]
Sep 11 17:44:35.541593 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:35.541571    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-83a2bc-openshift-default-79d84584-ddhbb"]
Sep 11 17:44:36.126651 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:36.126613    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="653eff13-fedd-469b-b0ac-e9d51e6da73b" path="/var/lib/kubelet/pods/653eff13-fedd-469b-b0ac-e9d51e6da73b/volumes"
Sep 11 17:44:40.335096 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:40.335051    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-290683-555989d749-5zcfd"]
Sep 11 17:44:40.335919 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:40.335860    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-290683-555989d749-5zcfd" podUID="05f0381c-b8e4-46d3-9046-888477b55704" containerName="payload-processing" containerID="cri-o://c06363fc49987cd3ac3797bb3b5be91cafc651c87785c8da2ec1d12d8f82a8e2" gracePeriod=30
Sep 11 17:44:40.343697 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:40.343669    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-290683-7546dcdd98-pnm98"]
Sep 11 17:44:40.344493 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:40.344466    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-290683-7546dcdd98-pnm98" podUID="8021f9fd-66b1-40ea-b776-2119913955d6" containerName="payload-pre-processing" containerID="cri-o://c92cc67474c8ec84b5f8dbb971402f27617b6f5ec6b1c67d8308a6c4e55bcd13" gracePeriod=30
Sep 11 17:44:40.544364 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:40.544274    2562 generic.go:356] "Generic (PLEG): container finished" podID="05f0381c-b8e4-46d3-9046-888477b55704" containerID="c06363fc49987cd3ac3797bb3b5be91cafc651c87785c8da2ec1d12d8f82a8e2" exitCode=0
Sep 11 17:44:40.544364 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:40.544325    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-290683-555989d749-5zcfd" event={"ID":"05f0381c-b8e4-46d3-9046-888477b55704","Type":"ContainerDied","Data":"c06363fc49987cd3ac3797bb3b5be91cafc651c87785c8da2ec1d12d8f82a8e2"}
Sep 11 17:44:40.547265 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:40.546316    2562 generic.go:356] "Generic (PLEG): container finished" podID="8021f9fd-66b1-40ea-b776-2119913955d6" containerID="c92cc67474c8ec84b5f8dbb971402f27617b6f5ec6b1c67d8308a6c4e55bcd13" exitCode=0
Sep 11 17:44:40.547265 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:40.546431    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-290683-7546dcdd98-pnm98" event={"ID":"8021f9fd-66b1-40ea-b776-2119913955d6","Type":"ContainerDied","Data":"c92cc67474c8ec84b5f8dbb971402f27617b6f5ec6b1c67d8308a6c4e55bcd13"}
Sep 11 17:44:40.613862 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:40.613836    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-290683-555989d749-5zcfd"
Sep 11 17:44:40.689842 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:40.689807    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-290683-7546dcdd98-pnm98"
Sep 11 17:44:40.786924 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:40.786881    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrl46\" (UniqueName: \"kubernetes.io/projected/05f0381c-b8e4-46d3-9046-888477b55704-kube-api-access-vrl46\") pod \"05f0381c-b8e4-46d3-9046-888477b55704\" (UID: \"05f0381c-b8e4-46d3-9046-888477b55704\") "
Sep 11 17:44:40.787146 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:40.787131    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05f0381c-b8e4-46d3-9046-888477b55704-plugins-config-volume\") pod \"05f0381c-b8e4-46d3-9046-888477b55704\" (UID: \"05f0381c-b8e4-46d3-9046-888477b55704\") "
Sep 11 17:44:40.787537 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:40.787505    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05f0381c-b8e4-46d3-9046-888477b55704-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "05f0381c-b8e4-46d3-9046-888477b55704" (UID: "05f0381c-b8e4-46d3-9046-888477b55704"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:44:40.789096 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:40.789068    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f0381c-b8e4-46d3-9046-888477b55704-kube-api-access-vrl46" (OuterVolumeSpecName: "kube-api-access-vrl46") pod "05f0381c-b8e4-46d3-9046-888477b55704" (UID: "05f0381c-b8e4-46d3-9046-888477b55704"). InnerVolumeSpecName "kube-api-access-vrl46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:44:40.887881 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:40.887846    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcldd\" (UniqueName: \"kubernetes.io/projected/8021f9fd-66b1-40ea-b776-2119913955d6-kube-api-access-lcldd\") pod \"8021f9fd-66b1-40ea-b776-2119913955d6\" (UID: \"8021f9fd-66b1-40ea-b776-2119913955d6\") "
Sep 11 17:44:40.888098 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:40.887953    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8021f9fd-66b1-40ea-b776-2119913955d6-plugins-config-volume\") pod \"8021f9fd-66b1-40ea-b776-2119913955d6\" (UID: \"8021f9fd-66b1-40ea-b776-2119913955d6\") "
Sep 11 17:44:40.888299 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:40.888277    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05f0381c-b8e4-46d3-9046-888477b55704-plugins-config-volume\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:44:40.888371 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:40.888303    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrl46\" (UniqueName: \"kubernetes.io/projected/05f0381c-b8e4-46d3-9046-888477b55704-kube-api-access-vrl46\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:44:40.888371 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:40.888345    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8021f9fd-66b1-40ea-b776-2119913955d6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8021f9fd-66b1-40ea-b776-2119913955d6" (UID: "8021f9fd-66b1-40ea-b776-2119913955d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:44:40.890017 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:40.889969    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8021f9fd-66b1-40ea-b776-2119913955d6-kube-api-access-lcldd" (OuterVolumeSpecName: "kube-api-access-lcldd") pod "8021f9fd-66b1-40ea-b776-2119913955d6" (UID: "8021f9fd-66b1-40ea-b776-2119913955d6"). InnerVolumeSpecName "kube-api-access-lcldd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:44:40.989033 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:40.988985    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcldd\" (UniqueName: \"kubernetes.io/projected/8021f9fd-66b1-40ea-b776-2119913955d6-kube-api-access-lcldd\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:44:40.989033 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:40.989031    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8021f9fd-66b1-40ea-b776-2119913955d6-plugins-config-volume\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:44:41.553533 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:41.553495    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-290683-555989d749-5zcfd" event={"ID":"05f0381c-b8e4-46d3-9046-888477b55704","Type":"ContainerDied","Data":"2fdd448870b3e718ddfe2be9e6c448687d6d41255c120f228e16886d2be70d53"}
Sep 11 17:44:41.553533 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:41.553512    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-290683-555989d749-5zcfd"
Sep 11 17:44:41.554075 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:41.553582    2562 scope.go:117] "RemoveContainer" containerID="c06363fc49987cd3ac3797bb3b5be91cafc651c87785c8da2ec1d12d8f82a8e2"
Sep 11 17:44:41.554958 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:41.554919    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-290683-7546dcdd98-pnm98" event={"ID":"8021f9fd-66b1-40ea-b776-2119913955d6","Type":"ContainerDied","Data":"1075f5a66d1841de6cbe8180b580b9b8813978e16e1ba8ec397c00c033fed6be"}
Sep 11 17:44:41.554958 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:41.554948    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-290683-7546dcdd98-pnm98"
Sep 11 17:44:41.564221 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:41.564206    2562 scope.go:117] "RemoveContainer" containerID="c92cc67474c8ec84b5f8dbb971402f27617b6f5ec6b1c67d8308a6c4e55bcd13"
Sep 11 17:44:41.577304 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:41.577281    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-290683-555989d749-5zcfd"]
Sep 11 17:44:41.582956 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:41.582936    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-290683-555989d749-5zcfd"]
Sep 11 17:44:41.600385 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:41.600357    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-290683-7546dcdd98-pnm98"]
Sep 11 17:44:41.609557 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:41.609487    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-290683-7546dcdd98-pnm98"]
Sep 11 17:44:42.132508 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:42.132477    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05f0381c-b8e4-46d3-9046-888477b55704" path="/var/lib/kubelet/pods/05f0381c-b8e4-46d3-9046-888477b55704/volumes"
Sep 11 17:44:42.133013 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:42.132982    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8021f9fd-66b1-40ea-b776-2119913955d6" path="/var/lib/kubelet/pods/8021f9fd-66b1-40ea-b776-2119913955d6/volumes"
Sep 11 17:44:46.794570 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:46.794519    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c7d192/distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9"]
Sep 11 17:44:57.519903 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:57.519870    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz"]
Sep 11 17:44:57.520506 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:57.520155    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz" podUID="465d1839-c244-466a-9dca-c9ff8a9ab66a" containerName="maas-api" containerID="cri-o://de7711a520970670a6a1bd1d54b15373ad9a8ddeb0fd68e8513e75a08651bd4f" gracePeriod=30
Sep 11 17:44:57.639615 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:57.639588    2562 generic.go:356] "Generic (PLEG): container finished" podID="465d1839-c244-466a-9dca-c9ff8a9ab66a" containerID="de7711a520970670a6a1bd1d54b15373ad9a8ddeb0fd68e8513e75a08651bd4f" exitCode=0
Sep 11 17:44:57.639721 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:57.639649    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz" event={"ID":"465d1839-c244-466a-9dca-c9ff8a9ab66a","Type":"ContainerDied","Data":"de7711a520970670a6a1bd1d54b15373ad9a8ddeb0fd68e8513e75a08651bd4f"}
Sep 11 17:44:57.762471 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:57.762447    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz"
Sep 11 17:44:57.853926 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:57.853892    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/465d1839-c244-466a-9dca-c9ff8a9ab66a-maas-api-tls\") pod \"465d1839-c244-466a-9dca-c9ff8a9ab66a\" (UID: \"465d1839-c244-466a-9dca-c9ff8a9ab66a\") "
Sep 11 17:44:57.854168 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:57.853959    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/465d1839-c244-466a-9dca-c9ff8a9ab66a-metrics-certs\") pod \"465d1839-c244-466a-9dca-c9ff8a9ab66a\" (UID: \"465d1839-c244-466a-9dca-c9ff8a9ab66a\") "
Sep 11 17:44:57.854168 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:57.854104    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/465d1839-c244-466a-9dca-c9ff8a9ab66a-openshift-service-ca\") pod \"465d1839-c244-466a-9dca-c9ff8a9ab66a\" (UID: \"465d1839-c244-466a-9dca-c9ff8a9ab66a\") "
Sep 11 17:44:57.854168 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:57.854141    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkrmq\" (UniqueName: \"kubernetes.io/projected/465d1839-c244-466a-9dca-c9ff8a9ab66a-kube-api-access-hkrmq\") pod \"465d1839-c244-466a-9dca-c9ff8a9ab66a\" (UID: \"465d1839-c244-466a-9dca-c9ff8a9ab66a\") "
Sep 11 17:44:57.854693 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:57.854662    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/465d1839-c244-466a-9dca-c9ff8a9ab66a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "465d1839-c244-466a-9dca-c9ff8a9ab66a" (UID: "465d1839-c244-466a-9dca-c9ff8a9ab66a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:44:57.856280 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:57.856243    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/465d1839-c244-466a-9dca-c9ff8a9ab66a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "465d1839-c244-466a-9dca-c9ff8a9ab66a" (UID: "465d1839-c244-466a-9dca-c9ff8a9ab66a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:44:57.856938 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:57.856906    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/465d1839-c244-466a-9dca-c9ff8a9ab66a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "465d1839-c244-466a-9dca-c9ff8a9ab66a" (UID: "465d1839-c244-466a-9dca-c9ff8a9ab66a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:44:57.857207 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:57.857181    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/465d1839-c244-466a-9dca-c9ff8a9ab66a-kube-api-access-hkrmq" (OuterVolumeSpecName: "kube-api-access-hkrmq") pod "465d1839-c244-466a-9dca-c9ff8a9ab66a" (UID: "465d1839-c244-466a-9dca-c9ff8a9ab66a"). InnerVolumeSpecName "kube-api-access-hkrmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:44:57.956061 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:57.956021    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/465d1839-c244-466a-9dca-c9ff8a9ab66a-openshift-service-ca\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:44:57.956061 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:57.956056    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkrmq\" (UniqueName: \"kubernetes.io/projected/465d1839-c244-466a-9dca-c9ff8a9ab66a-kube-api-access-hkrmq\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:44:57.956061 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:57.956067    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/465d1839-c244-466a-9dca-c9ff8a9ab66a-maas-api-tls\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:44:57.956358 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:57.956076    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/465d1839-c244-466a-9dca-c9ff8a9ab66a-metrics-certs\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:44:58.646720 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:58.646690    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz"
Sep 11 17:44:58.646720 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:58.646712    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz" event={"ID":"465d1839-c244-466a-9dca-c9ff8a9ab66a","Type":"ContainerDied","Data":"18fe0ef77efc9142363adabf89441771caa8d0157334fc5ebbeda4de2dc6a544"}
Sep 11 17:44:58.647308 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:58.646774    2562 scope.go:117] "RemoveContainer" containerID="de7711a520970670a6a1bd1d54b15373ad9a8ddeb0fd68e8513e75a08651bd4f"
Sep 11 17:44:58.673664 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:58.673631    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz"]
Sep 11 17:44:58.679274 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:44:58.679248    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c7d192-58d88f8f58-dlcgz"]
Sep 11 17:45:00.099918 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.099884    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-2f0a78-59bd4ff499-qcqkn"]
Sep 11 17:45:00.100428 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.100111    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2f0a78-59bd4ff499-qcqkn" podUID="39cd2308-d436-41ca-b9e8-02078533374e" containerName="payload-processing" containerID="cri-o://13a22d95b8fcfedd976f6f420fc73cad833e8ea2fe1f8cf7dbca0137b5c50d16" gracePeriod=30
Sep 11 17:45:00.117139 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.116887    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2f0a78-765f68f474-88w8m"]
Sep 11 17:45:00.117139 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.117110    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2f0a78-765f68f474-88w8m" podUID="f64a7c90-1859-4bb7-afa8-76b8c51f2527" containerName="payload-pre-processing" containerID="cri-o://881ca0fcb951782a8282ae59e2d983ac509b92287122e1497b52423162098a4a" gracePeriod=30
Sep 11 17:45:00.130038 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.129986    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="465d1839-c244-466a-9dca-c9ff8a9ab66a" path="/var/lib/kubelet/pods/465d1839-c244-466a-9dca-c9ff8a9ab66a/volumes"
Sep 11 17:45:00.158028 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.157980    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fac54-29819145-zjh8f"]
Sep 11 17:45:00.158386 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.158373    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="653eff13-fedd-469b-b0ac-e9d51e6da73b" containerName="istio-proxy"
Sep 11 17:45:00.158432 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.158387    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="653eff13-fedd-469b-b0ac-e9d51e6da73b" containerName="istio-proxy"
Sep 11 17:45:00.158432 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.158404    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05f0381c-b8e4-46d3-9046-888477b55704" containerName="payload-processing"
Sep 11 17:45:00.158432 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.158410    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="05f0381c-b8e4-46d3-9046-888477b55704" containerName="payload-processing"
Sep 11 17:45:00.158432 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.158419    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="653eff13-fedd-469b-b0ac-e9d51e6da73b" containerName="istio-proxy"
Sep 11 17:45:00.158432 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.158425    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="653eff13-fedd-469b-b0ac-e9d51e6da73b" containerName="istio-proxy"
Sep 11 17:45:00.158589 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.158436    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="465d1839-c244-466a-9dca-c9ff8a9ab66a" containerName="maas-api"
Sep 11 17:45:00.158589 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.158443    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="465d1839-c244-466a-9dca-c9ff8a9ab66a" containerName="maas-api"
Sep 11 17:45:00.158589 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.158453    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8021f9fd-66b1-40ea-b776-2119913955d6" containerName="payload-pre-processing"
Sep 11 17:45:00.158589 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.158459    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8021f9fd-66b1-40ea-b776-2119913955d6" containerName="payload-pre-processing"
Sep 11 17:45:00.158589 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.158466    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34712b21-424b-495f-a78a-e42d802483de" containerName="payload-processing"
Sep 11 17:45:00.158589 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.158471    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="34712b21-424b-495f-a78a-e42d802483de" containerName="payload-processing"
Sep 11 17:45:00.158589 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.158522    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8021f9fd-66b1-40ea-b776-2119913955d6" containerName="payload-pre-processing"
Sep 11 17:45:00.158589 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.158528    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="34712b21-424b-495f-a78a-e42d802483de" containerName="payload-processing"
Sep 11 17:45:00.158589 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.158536    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="05f0381c-b8e4-46d3-9046-888477b55704" containerName="payload-processing"
Sep 11 17:45:00.158589 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.158543    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="653eff13-fedd-469b-b0ac-e9d51e6da73b" containerName="istio-proxy"
Sep 11 17:45:00.158589 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.158549    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="653eff13-fedd-469b-b0ac-e9d51e6da73b" containerName="istio-proxy"
Sep 11 17:45:00.158589 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.158555    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="465d1839-c244-466a-9dca-c9ff8a9ab66a" containerName="maas-api"
Sep 11 17:45:00.161696 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.161675    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fac54-29819145-zjh8f"
Sep 11 17:45:00.164225 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.164207    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9n8gf\""
Sep 11 17:45:00.185253 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.185222    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fac54-29819145-zjh8f"]
Sep 11 17:45:00.234102 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.234073    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fac54-29819145-zjh8f"
Sep 11 17:45:00.398439 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.398408    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2f0a78-59bd4ff499-qcqkn"
Sep 11 17:45:00.427493 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.427466    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2f0a78-765f68f474-88w8m"
Sep 11 17:45:00.482675 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.482636    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4bjt\" (UniqueName: \"kubernetes.io/projected/39cd2308-d436-41ca-b9e8-02078533374e-kube-api-access-s4bjt\") pod \"39cd2308-d436-41ca-b9e8-02078533374e\" (UID: \"39cd2308-d436-41ca-b9e8-02078533374e\") "
Sep 11 17:45:00.482887 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.482728    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wch29\" (UniqueName: \"kubernetes.io/projected/f64a7c90-1859-4bb7-afa8-76b8c51f2527-kube-api-access-wch29\") pod \"f64a7c90-1859-4bb7-afa8-76b8c51f2527\" (UID: \"f64a7c90-1859-4bb7-afa8-76b8c51f2527\") "
Sep 11 17:45:00.482887 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.482764    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39cd2308-d436-41ca-b9e8-02078533374e-plugins-config-volume\") pod \"39cd2308-d436-41ca-b9e8-02078533374e\" (UID: \"39cd2308-d436-41ca-b9e8-02078533374e\") "
Sep 11 17:45:00.482887 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.482804    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f64a7c90-1859-4bb7-afa8-76b8c51f2527-plugins-config-volume\") pod \"f64a7c90-1859-4bb7-afa8-76b8c51f2527\" (UID: \"f64a7c90-1859-4bb7-afa8-76b8c51f2527\") "
Sep 11 17:45:00.483557 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.483528    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f64a7c90-1859-4bb7-afa8-76b8c51f2527-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f64a7c90-1859-4bb7-afa8-76b8c51f2527" (UID: "f64a7c90-1859-4bb7-afa8-76b8c51f2527"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:45:00.483777 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.483753    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39cd2308-d436-41ca-b9e8-02078533374e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "39cd2308-d436-41ca-b9e8-02078533374e" (UID: "39cd2308-d436-41ca-b9e8-02078533374e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:45:00.485715 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.485673    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39cd2308-d436-41ca-b9e8-02078533374e-kube-api-access-s4bjt" (OuterVolumeSpecName: "kube-api-access-s4bjt") pod "39cd2308-d436-41ca-b9e8-02078533374e" (UID: "39cd2308-d436-41ca-b9e8-02078533374e"). InnerVolumeSpecName "kube-api-access-s4bjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:45:00.486023 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.485879    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f64a7c90-1859-4bb7-afa8-76b8c51f2527-kube-api-access-wch29" (OuterVolumeSpecName: "kube-api-access-wch29") pod "f64a7c90-1859-4bb7-afa8-76b8c51f2527" (UID: "f64a7c90-1859-4bb7-afa8-76b8c51f2527"). InnerVolumeSpecName "kube-api-access-wch29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:45:00.519132 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.519093    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fac54-29819145-zjh8f"]
Sep 11 17:45:00.522623 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:45:00.522591    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c8990a4_279f_40a5_bf70_a148874ff90f.slice/crio-6366d73d2cc54fe4be217e0eddd01e5b474f1d08728b9b5d3258dba791ba198e WatchSource:0}: Error finding container 6366d73d2cc54fe4be217e0eddd01e5b474f1d08728b9b5d3258dba791ba198e: Status 404 returned error can't find the container with id 6366d73d2cc54fe4be217e0eddd01e5b474f1d08728b9b5d3258dba791ba198e
Sep 11 17:45:00.524521 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.524504    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 17:45:00.583785 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.583750    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wch29\" (UniqueName: \"kubernetes.io/projected/f64a7c90-1859-4bb7-afa8-76b8c51f2527-kube-api-access-wch29\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:45:00.583785 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.583778    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39cd2308-d436-41ca-b9e8-02078533374e-plugins-config-volume\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:45:00.583785 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.583789    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f64a7c90-1859-4bb7-afa8-76b8c51f2527-plugins-config-volume\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:45:00.584067 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.583798    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4bjt\" (UniqueName: \"kubernetes.io/projected/39cd2308-d436-41ca-b9e8-02078533374e-kube-api-access-s4bjt\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:45:00.661311 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.661216    2562 generic.go:356] "Generic (PLEG): container finished" podID="39cd2308-d436-41ca-b9e8-02078533374e" containerID="13a22d95b8fcfedd976f6f420fc73cad833e8ea2fe1f8cf7dbca0137b5c50d16" exitCode=0
Sep 11 17:45:00.661311 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.661264    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2f0a78-59bd4ff499-qcqkn"
Sep 11 17:45:00.661311 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.661302    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2f0a78-59bd4ff499-qcqkn" event={"ID":"39cd2308-d436-41ca-b9e8-02078533374e","Type":"ContainerDied","Data":"13a22d95b8fcfedd976f6f420fc73cad833e8ea2fe1f8cf7dbca0137b5c50d16"}
Sep 11 17:45:00.661584 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.661355    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2f0a78-59bd4ff499-qcqkn" event={"ID":"39cd2308-d436-41ca-b9e8-02078533374e","Type":"ContainerDied","Data":"1e45a9132807ce657195840016e48266519af65fe6cf63191afced5c2e5138f1"}
Sep 11 17:45:00.661584 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.661378    2562 scope.go:117] "RemoveContainer" containerID="13a22d95b8fcfedd976f6f420fc73cad833e8ea2fe1f8cf7dbca0137b5c50d16"
Sep 11 17:45:00.662957 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.662929    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fac54-29819145-zjh8f" event={"ID":"4c8990a4-279f-40a5-bf70-a148874ff90f","Type":"ContainerStarted","Data":"6366d73d2cc54fe4be217e0eddd01e5b474f1d08728b9b5d3258dba791ba198e"}
Sep 11 17:45:00.664349 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.664329    2562 generic.go:356] "Generic (PLEG): container finished" podID="f64a7c90-1859-4bb7-afa8-76b8c51f2527" containerID="881ca0fcb951782a8282ae59e2d983ac509b92287122e1497b52423162098a4a" exitCode=0
Sep 11 17:45:00.664467 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.664376    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2f0a78-765f68f474-88w8m"
Sep 11 17:45:00.664467 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.664408    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2f0a78-765f68f474-88w8m" event={"ID":"f64a7c90-1859-4bb7-afa8-76b8c51f2527","Type":"ContainerDied","Data":"881ca0fcb951782a8282ae59e2d983ac509b92287122e1497b52423162098a4a"}
Sep 11 17:45:00.664467 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.664458    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2f0a78-765f68f474-88w8m" event={"ID":"f64a7c90-1859-4bb7-afa8-76b8c51f2527","Type":"ContainerDied","Data":"392c129751cc0c3a4d1bda402d3ead615497efc5aff2fba595ed2d6b703a79c1"}
Sep 11 17:45:00.672347 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.672233    2562 scope.go:117] "RemoveContainer" containerID="13a22d95b8fcfedd976f6f420fc73cad833e8ea2fe1f8cf7dbca0137b5c50d16"
Sep 11 17:45:00.673514 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:45:00.672836    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13a22d95b8fcfedd976f6f420fc73cad833e8ea2fe1f8cf7dbca0137b5c50d16\": container with ID starting with 13a22d95b8fcfedd976f6f420fc73cad833e8ea2fe1f8cf7dbca0137b5c50d16 not found: ID does not exist" containerID="13a22d95b8fcfedd976f6f420fc73cad833e8ea2fe1f8cf7dbca0137b5c50d16"
Sep 11 17:45:00.673514 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.672886    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13a22d95b8fcfedd976f6f420fc73cad833e8ea2fe1f8cf7dbca0137b5c50d16"} err="failed to get container status \"13a22d95b8fcfedd976f6f420fc73cad833e8ea2fe1f8cf7dbca0137b5c50d16\": rpc error: code = NotFound desc = could not find container \"13a22d95b8fcfedd976f6f420fc73cad833e8ea2fe1f8cf7dbca0137b5c50d16\": container with ID starting with 13a22d95b8fcfedd976f6f420fc73cad833e8ea2fe1f8cf7dbca0137b5c50d16 not found: ID does not exist"
Sep 11 17:45:00.673514 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.672911    2562 scope.go:117] "RemoveContainer" containerID="881ca0fcb951782a8282ae59e2d983ac509b92287122e1497b52423162098a4a"
Sep 11 17:45:00.695231 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.695198    2562 scope.go:117] "RemoveContainer" containerID="881ca0fcb951782a8282ae59e2d983ac509b92287122e1497b52423162098a4a"
Sep 11 17:45:00.695404 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.695385    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-2f0a78-59bd4ff499-qcqkn"]
Sep 11 17:45:00.695825 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:45:00.695783    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"881ca0fcb951782a8282ae59e2d983ac509b92287122e1497b52423162098a4a\": container with ID starting with 881ca0fcb951782a8282ae59e2d983ac509b92287122e1497b52423162098a4a not found: ID does not exist" containerID="881ca0fcb951782a8282ae59e2d983ac509b92287122e1497b52423162098a4a"
Sep 11 17:45:00.695951 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.695831    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"881ca0fcb951782a8282ae59e2d983ac509b92287122e1497b52423162098a4a"} err="failed to get container status \"881ca0fcb951782a8282ae59e2d983ac509b92287122e1497b52423162098a4a\": rpc error: code = NotFound desc = could not find container \"881ca0fcb951782a8282ae59e2d983ac509b92287122e1497b52423162098a4a\": container with ID starting with 881ca0fcb951782a8282ae59e2d983ac509b92287122e1497b52423162098a4a not found: ID does not exist"
Sep 11 17:45:00.697647 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.697623    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-2f0a78-59bd4ff499-qcqkn"]
Sep 11 17:45:00.716326 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.716288    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2f0a78-765f68f474-88w8m"]
Sep 11 17:45:00.719176 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:00.719149    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2f0a78-765f68f474-88w8m"]
Sep 11 17:45:01.814011 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:01.813954    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-c7d192/distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9" podUID="fcef6d19-ab56-4c36-8725-ce2253241999" containerName="main" containerID="cri-o://b341462a694a05c501a5f46d870a8587fb8e62074219669923f4e2165d773abd" gracePeriod=45
Sep 11 17:45:02.085144 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:02.085114    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c7d192/distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9"
Sep 11 17:45:02.129776 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:02.129722    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39cd2308-d436-41ca-b9e8-02078533374e" path="/var/lib/kubelet/pods/39cd2308-d436-41ca-b9e8-02078533374e/volumes"
Sep 11 17:45:02.130283 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:02.130264    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f64a7c90-1859-4bb7-afa8-76b8c51f2527" path="/var/lib/kubelet/pods/f64a7c90-1859-4bb7-afa8-76b8c51f2527/volumes"
Sep 11 17:45:02.201502 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:02.200952    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcef6d19-ab56-4c36-8725-ce2253241999-tmp-dir\") pod \"fcef6d19-ab56-4c36-8725-ce2253241999\" (UID: \"fcef6d19-ab56-4c36-8725-ce2253241999\") "
Sep 11 17:45:02.201502 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:02.201047    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcef6d19-ab56-4c36-8725-ce2253241999-home\") pod \"fcef6d19-ab56-4c36-8725-ce2253241999\" (UID: \"fcef6d19-ab56-4c36-8725-ce2253241999\") "
Sep 11 17:45:02.201502 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:02.201128    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcef6d19-ab56-4c36-8725-ce2253241999-model-cache\") pod \"fcef6d19-ab56-4c36-8725-ce2253241999\" (UID: \"fcef6d19-ab56-4c36-8725-ce2253241999\") "
Sep 11 17:45:02.201502 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:02.201180    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d78vz\" (UniqueName: \"kubernetes.io/projected/fcef6d19-ab56-4c36-8725-ce2253241999-kube-api-access-d78vz\") pod \"fcef6d19-ab56-4c36-8725-ce2253241999\" (UID: \"fcef6d19-ab56-4c36-8725-ce2253241999\") "
Sep 11 17:45:02.201502 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:02.201230    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcef6d19-ab56-4c36-8725-ce2253241999-dshm\") pod \"fcef6d19-ab56-4c36-8725-ce2253241999\" (UID: \"fcef6d19-ab56-4c36-8725-ce2253241999\") "
Sep 11 17:45:02.201502 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:02.201304    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcef6d19-ab56-4c36-8725-ce2253241999-tls-certs\") pod \"fcef6d19-ab56-4c36-8725-ce2253241999\" (UID: \"fcef6d19-ab56-4c36-8725-ce2253241999\") "
Sep 11 17:45:02.202017 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:02.201818    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcef6d19-ab56-4c36-8725-ce2253241999-home" (OuterVolumeSpecName: "home") pod "fcef6d19-ab56-4c36-8725-ce2253241999" (UID: "fcef6d19-ab56-4c36-8725-ce2253241999"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:02.202516 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:02.202470    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcef6d19-ab56-4c36-8725-ce2253241999-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fcef6d19-ab56-4c36-8725-ce2253241999" (UID: "fcef6d19-ab56-4c36-8725-ce2253241999"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:02.203015 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:02.202692    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcef6d19-ab56-4c36-8725-ce2253241999-model-cache" (OuterVolumeSpecName: "model-cache") pod "fcef6d19-ab56-4c36-8725-ce2253241999" (UID: "fcef6d19-ab56-4c36-8725-ce2253241999"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:02.204229 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:02.204150    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcef6d19-ab56-4c36-8725-ce2253241999-tmp-dir\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:45:02.204229 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:02.204194    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcef6d19-ab56-4c36-8725-ce2253241999-home\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:45:02.204229 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:02.204208    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcef6d19-ab56-4c36-8725-ce2253241999-model-cache\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:45:02.206091 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:02.206050    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcef6d19-ab56-4c36-8725-ce2253241999-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fcef6d19-ab56-4c36-8725-ce2253241999" (UID: "fcef6d19-ab56-4c36-8725-ce2253241999"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:45:02.206091 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:02.206067    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcef6d19-ab56-4c36-8725-ce2253241999-dshm" (OuterVolumeSpecName: "dshm") pod "fcef6d19-ab56-4c36-8725-ce2253241999" (UID: "fcef6d19-ab56-4c36-8725-ce2253241999"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:02.206255 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:02.206109    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcef6d19-ab56-4c36-8725-ce2253241999-kube-api-access-d78vz" (OuterVolumeSpecName: "kube-api-access-d78vz") pod "fcef6d19-ab56-4c36-8725-ce2253241999" (UID: "fcef6d19-ab56-4c36-8725-ce2253241999"). InnerVolumeSpecName "kube-api-access-d78vz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:45:02.305071 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:02.305028    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcef6d19-ab56-4c36-8725-ce2253241999-tls-certs\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:45:02.305071 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:02.305072    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d78vz\" (UniqueName: \"kubernetes.io/projected/fcef6d19-ab56-4c36-8725-ce2253241999-kube-api-access-d78vz\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:45:02.305326 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:02.305088    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcef6d19-ab56-4c36-8725-ce2253241999-dshm\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:45:02.681629 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:02.681592    2562 generic.go:356] "Generic (PLEG): container finished" podID="fcef6d19-ab56-4c36-8725-ce2253241999" containerID="b341462a694a05c501a5f46d870a8587fb8e62074219669923f4e2165d773abd" exitCode=0
Sep 11 17:45:02.681822 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:02.681660    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c7d192/distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9"
Sep 11 17:45:02.681822 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:02.681661    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c7d192/distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9" event={"ID":"fcef6d19-ab56-4c36-8725-ce2253241999","Type":"ContainerDied","Data":"b341462a694a05c501a5f46d870a8587fb8e62074219669923f4e2165d773abd"}
Sep 11 17:45:02.681822 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:02.681782    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c7d192/distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9" event={"ID":"fcef6d19-ab56-4c36-8725-ce2253241999","Type":"ContainerDied","Data":"c763d6e554c8f2a34ab726c721e729596bfba9e0cf632a8ffcfd005350b6f629"}
Sep 11 17:45:02.681822 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:02.681803    2562 scope.go:117] "RemoveContainer" containerID="b341462a694a05c501a5f46d870a8587fb8e62074219669923f4e2165d773abd"
Sep 11 17:45:02.709554 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:02.709503    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c7d192/distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9"]
Sep 11 17:45:02.718568 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:02.718530    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-c7d192/distinct-2-w0-c7d192-kserve-5d6dbf5478-pr9f9"]
Sep 11 17:45:02.805432 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:02.805408    2562 scope.go:117] "RemoveContainer" containerID="b341462a694a05c501a5f46d870a8587fb8e62074219669923f4e2165d773abd"
Sep 11 17:45:02.805808 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:45:02.805787    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b341462a694a05c501a5f46d870a8587fb8e62074219669923f4e2165d773abd\": container with ID starting with b341462a694a05c501a5f46d870a8587fb8e62074219669923f4e2165d773abd not found: ID does not exist" containerID="b341462a694a05c501a5f46d870a8587fb8e62074219669923f4e2165d773abd"
Sep 11 17:45:02.805916 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:02.805817    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b341462a694a05c501a5f46d870a8587fb8e62074219669923f4e2165d773abd"} err="failed to get container status \"b341462a694a05c501a5f46d870a8587fb8e62074219669923f4e2165d773abd\": rpc error: code = NotFound desc = could not find container \"b341462a694a05c501a5f46d870a8587fb8e62074219669923f4e2165d773abd\": container with ID starting with b341462a694a05c501a5f46d870a8587fb8e62074219669923f4e2165d773abd not found: ID does not exist"
Sep 11 17:45:03.688882 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:03.688839    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fac54-29819145-zjh8f" event={"ID":"4c8990a4-279f-40a5-bf70-a148874ff90f","Type":"ContainerStarted","Data":"18995f2c1ae77c9a47884adb47dbc5d3d44f15b454ed2ac8b501bbebc46c1d88"}
Sep 11 17:45:04.126303 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:04.126264    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcef6d19-ab56-4c36-8725-ce2253241999" path="/var/lib/kubelet/pods/fcef6d19-ab56-4c36-8725-ce2253241999/volumes"
Sep 11 17:45:04.701963 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:04.701927    2562 generic.go:356] "Generic (PLEG): container finished" podID="4c8990a4-279f-40a5-bf70-a148874ff90f" containerID="18995f2c1ae77c9a47884adb47dbc5d3d44f15b454ed2ac8b501bbebc46c1d88" exitCode=0
Sep 11 17:45:04.702401 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:04.702023    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fac54-29819145-zjh8f" event={"ID":"4c8990a4-279f-40a5-bf70-a148874ff90f","Type":"ContainerDied","Data":"18995f2c1ae77c9a47884adb47dbc5d3d44f15b454ed2ac8b501bbebc46c1d88"}
Sep 11 17:45:05.846474 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:05.846446    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fac54-29819145-zjh8f"
Sep 11 17:45:06.713196 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:06.713159    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fac54-29819145-zjh8f" event={"ID":"4c8990a4-279f-40a5-bf70-a148874ff90f","Type":"ContainerDied","Data":"6366d73d2cc54fe4be217e0eddd01e5b474f1d08728b9b5d3258dba791ba198e"}
Sep 11 17:45:06.713196 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:06.713180    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fac54-29819145-zjh8f"
Sep 11 17:45:06.713469 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:06.713210    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6366d73d2cc54fe4be217e0eddd01e5b474f1d08728b9b5d3258dba791ba198e"
Sep 11 17:45:07.549945 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:07.549913    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"]
Sep 11 17:45:07.550387 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:07.550124    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4" podUID="00b26df4-2ff6-45f3-b020-e1854b2060d6" containerName="istio-proxy" containerID="cri-o://6476d3b8dc311012ba5147baaa84da12977b4c6f9e384d3472b6e970611bfbc8" gracePeriod=30
Sep 11 17:45:08.889069 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:08.889032    2562 patch_prober.go:28] interesting pod/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 17:45:08.889436 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:08.889094    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4" podUID="00b26df4-2ff6-45f3-b020-e1854b2060d6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 17:45:12.745844 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:12.745806    2562 generic.go:356] "Generic (PLEG): container finished" podID="00b26df4-2ff6-45f3-b020-e1854b2060d6" containerID="6476d3b8dc311012ba5147baaa84da12977b4c6f9e384d3472b6e970611bfbc8" exitCode=0
Sep 11 17:45:12.746248 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:12.745879    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4" event={"ID":"00b26df4-2ff6-45f3-b020-e1854b2060d6","Type":"ContainerDied","Data":"6476d3b8dc311012ba5147baaa84da12977b4c6f9e384d3472b6e970611bfbc8"}
Sep 11 17:45:12.746248 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:12.745946    2562 scope.go:117] "RemoveContainer" containerID="22f018c6731d122966eae74773279a0812791889fc4c6b848a8f54a8a32d1332"
Sep 11 17:45:12.803414 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:12.803390    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"
Sep 11 17:45:12.906020 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:12.905948    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00b26df4-2ff6-45f3-b020-e1854b2060d6-istio-token\") pod \"00b26df4-2ff6-45f3-b020-e1854b2060d6\" (UID: \"00b26df4-2ff6-45f3-b020-e1854b2060d6\") "
Sep 11 17:45:12.906217 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:12.906039    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00b26df4-2ff6-45f3-b020-e1854b2060d6-istio-data\") pod \"00b26df4-2ff6-45f3-b020-e1854b2060d6\" (UID: \"00b26df4-2ff6-45f3-b020-e1854b2060d6\") "
Sep 11 17:45:12.906217 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:12.906087    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8mzp\" (UniqueName: \"kubernetes.io/projected/00b26df4-2ff6-45f3-b020-e1854b2060d6-kube-api-access-l8mzp\") pod \"00b26df4-2ff6-45f3-b020-e1854b2060d6\" (UID: \"00b26df4-2ff6-45f3-b020-e1854b2060d6\") "
Sep 11 17:45:12.906217 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:12.906111    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00b26df4-2ff6-45f3-b020-e1854b2060d6-istio-podinfo\") pod \"00b26df4-2ff6-45f3-b020-e1854b2060d6\" (UID: \"00b26df4-2ff6-45f3-b020-e1854b2060d6\") "
Sep 11 17:45:12.906217 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:12.906141    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b26df4-2ff6-45f3-b020-e1854b2060d6-workload-socket\") pod \"00b26df4-2ff6-45f3-b020-e1854b2060d6\" (UID: \"00b26df4-2ff6-45f3-b020-e1854b2060d6\") "
Sep 11 17:45:12.906217 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:12.906171    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00b26df4-2ff6-45f3-b020-e1854b2060d6-workload-certs\") pod \"00b26df4-2ff6-45f3-b020-e1854b2060d6\" (UID: \"00b26df4-2ff6-45f3-b020-e1854b2060d6\") "
Sep 11 17:45:12.906217 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:12.906194    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b26df4-2ff6-45f3-b020-e1854b2060d6-credential-socket\") pod \"00b26df4-2ff6-45f3-b020-e1854b2060d6\" (UID: \"00b26df4-2ff6-45f3-b020-e1854b2060d6\") "
Sep 11 17:45:12.906541 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:12.906256    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00b26df4-2ff6-45f3-b020-e1854b2060d6-istio-envoy\") pod \"00b26df4-2ff6-45f3-b020-e1854b2060d6\" (UID: \"00b26df4-2ff6-45f3-b020-e1854b2060d6\") "
Sep 11 17:45:12.906541 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:12.906297    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00b26df4-2ff6-45f3-b020-e1854b2060d6-istiod-ca-cert\") pod \"00b26df4-2ff6-45f3-b020-e1854b2060d6\" (UID: \"00b26df4-2ff6-45f3-b020-e1854b2060d6\") "
Sep 11 17:45:12.906541 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:12.906418    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b26df4-2ff6-45f3-b020-e1854b2060d6-istio-data" (OuterVolumeSpecName: "istio-data") pod "00b26df4-2ff6-45f3-b020-e1854b2060d6" (UID: "00b26df4-2ff6-45f3-b020-e1854b2060d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:12.906700 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:12.906561    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b26df4-2ff6-45f3-b020-e1854b2060d6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "00b26df4-2ff6-45f3-b020-e1854b2060d6" (UID: "00b26df4-2ff6-45f3-b020-e1854b2060d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:12.906700 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:12.906658    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b26df4-2ff6-45f3-b020-e1854b2060d6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "00b26df4-2ff6-45f3-b020-e1854b2060d6" (UID: "00b26df4-2ff6-45f3-b020-e1854b2060d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:12.906796 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:12.906710    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b26df4-2ff6-45f3-b020-e1854b2060d6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "00b26df4-2ff6-45f3-b020-e1854b2060d6" (UID: "00b26df4-2ff6-45f3-b020-e1854b2060d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:12.906796 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:12.906767    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00b26df4-2ff6-45f3-b020-e1854b2060d6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "00b26df4-2ff6-45f3-b020-e1854b2060d6" (UID: "00b26df4-2ff6-45f3-b020-e1854b2060d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:45:12.906894 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:12.906811    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00b26df4-2ff6-45f3-b020-e1854b2060d6-istio-data\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:45:12.906894 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:12.906832    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00b26df4-2ff6-45f3-b020-e1854b2060d6-workload-certs\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:45:12.908325 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:12.908300    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00b26df4-2ff6-45f3-b020-e1854b2060d6-istio-token" (OuterVolumeSpecName: "istio-token") pod "00b26df4-2ff6-45f3-b020-e1854b2060d6" (UID: "00b26df4-2ff6-45f3-b020-e1854b2060d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:45:12.908413 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:12.908384    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00b26df4-2ff6-45f3-b020-e1854b2060d6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "00b26df4-2ff6-45f3-b020-e1854b2060d6" (UID: "00b26df4-2ff6-45f3-b020-e1854b2060d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 17:45:12.908669 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:12.908652    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00b26df4-2ff6-45f3-b020-e1854b2060d6-kube-api-access-l8mzp" (OuterVolumeSpecName: "kube-api-access-l8mzp") pod "00b26df4-2ff6-45f3-b020-e1854b2060d6" (UID: "00b26df4-2ff6-45f3-b020-e1854b2060d6"). InnerVolumeSpecName "kube-api-access-l8mzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:45:12.908730 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:12.908711    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b26df4-2ff6-45f3-b020-e1854b2060d6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "00b26df4-2ff6-45f3-b020-e1854b2060d6" (UID: "00b26df4-2ff6-45f3-b020-e1854b2060d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:13.007946 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:13.007909    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00b26df4-2ff6-45f3-b020-e1854b2060d6-istio-token\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:45:13.007946 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:13.007941    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8mzp\" (UniqueName: \"kubernetes.io/projected/00b26df4-2ff6-45f3-b020-e1854b2060d6-kube-api-access-l8mzp\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:45:13.007946 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:13.007952    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00b26df4-2ff6-45f3-b020-e1854b2060d6-istio-podinfo\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:45:13.008223 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:13.007961    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b26df4-2ff6-45f3-b020-e1854b2060d6-workload-socket\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:45:13.008223 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:13.007970    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b26df4-2ff6-45f3-b020-e1854b2060d6-credential-socket\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:45:13.008223 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:13.007978    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00b26df4-2ff6-45f3-b020-e1854b2060d6-istio-envoy\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:45:13.008223 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:13.007986    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00b26df4-2ff6-45f3-b020-e1854b2060d6-istiod-ca-cert\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:45:13.752923 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:13.752891    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"
Sep 11 17:45:13.752923 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:13.752903    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4" event={"ID":"00b26df4-2ff6-45f3-b020-e1854b2060d6","Type":"ContainerDied","Data":"d1518d53fb8dbf3f272fd7976ef6657be64e269a18e76ff322316c4fccaf4d0c"}
Sep 11 17:45:13.753456 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:13.752955    2562 scope.go:117] "RemoveContainer" containerID="6476d3b8dc311012ba5147baaa84da12977b4c6f9e384d3472b6e970611bfbc8"
Sep 11 17:45:13.774817 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:13.774783    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"]
Sep 11 17:45:13.778949 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:13.778922    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c7d192-openshift-default-7d5d48f4bd-dhqp4"]
Sep 11 17:45:14.125920 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:14.125890    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00b26df4-2ff6-45f3-b020-e1854b2060d6" path="/var/lib/kubelet/pods/00b26df4-2ff6-45f3-b020-e1854b2060d6/volumes"
Sep 11 17:45:43.891980 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:43.891941    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq"]
Sep 11 17:45:43.892519 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:43.892212    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq" podUID="dd26d755-6e39-49b6-b956-98d5709b58b0" containerName="maas-api" containerID="cri-o://ade2848bc4fb1509e4ba34f9270f68bafdaf92e451434a833ccb5987ee424d6e" gracePeriod=30
Sep 11 17:45:44.162358 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:44.162333    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq"
Sep 11 17:45:44.300981 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:44.300946    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62kfm\" (UniqueName: \"kubernetes.io/projected/dd26d755-6e39-49b6-b956-98d5709b58b0-kube-api-access-62kfm\") pod \"dd26d755-6e39-49b6-b956-98d5709b58b0\" (UID: \"dd26d755-6e39-49b6-b956-98d5709b58b0\") "
Sep 11 17:45:44.301174 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:44.300987    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd26d755-6e39-49b6-b956-98d5709b58b0-metrics-certs\") pod \"dd26d755-6e39-49b6-b956-98d5709b58b0\" (UID: \"dd26d755-6e39-49b6-b956-98d5709b58b0\") "
Sep 11 17:45:44.301174 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:44.301100    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd26d755-6e39-49b6-b956-98d5709b58b0-openshift-service-ca\") pod \"dd26d755-6e39-49b6-b956-98d5709b58b0\" (UID: \"dd26d755-6e39-49b6-b956-98d5709b58b0\") "
Sep 11 17:45:44.301174 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:44.301141    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd26d755-6e39-49b6-b956-98d5709b58b0-maas-api-tls\") pod \"dd26d755-6e39-49b6-b956-98d5709b58b0\" (UID: \"dd26d755-6e39-49b6-b956-98d5709b58b0\") "
Sep 11 17:45:44.301509 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:44.301484    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd26d755-6e39-49b6-b956-98d5709b58b0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dd26d755-6e39-49b6-b956-98d5709b58b0" (UID: "dd26d755-6e39-49b6-b956-98d5709b58b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:45:44.303299 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:44.303264    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd26d755-6e39-49b6-b956-98d5709b58b0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dd26d755-6e39-49b6-b956-98d5709b58b0" (UID: "dd26d755-6e39-49b6-b956-98d5709b58b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:45:44.303299 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:44.303285    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd26d755-6e39-49b6-b956-98d5709b58b0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "dd26d755-6e39-49b6-b956-98d5709b58b0" (UID: "dd26d755-6e39-49b6-b956-98d5709b58b0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:45:44.303462 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:44.303285    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd26d755-6e39-49b6-b956-98d5709b58b0-kube-api-access-62kfm" (OuterVolumeSpecName: "kube-api-access-62kfm") pod "dd26d755-6e39-49b6-b956-98d5709b58b0" (UID: "dd26d755-6e39-49b6-b956-98d5709b58b0"). InnerVolumeSpecName "kube-api-access-62kfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:45:44.403982 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:44.403832    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd26d755-6e39-49b6-b956-98d5709b58b0-maas-api-tls\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:45:44.403982 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:44.403867    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62kfm\" (UniqueName: \"kubernetes.io/projected/dd26d755-6e39-49b6-b956-98d5709b58b0-kube-api-access-62kfm\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:45:44.403982 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:44.403891    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd26d755-6e39-49b6-b956-98d5709b58b0-metrics-certs\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:45:44.403982 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:44.403905    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd26d755-6e39-49b6-b956-98d5709b58b0-openshift-service-ca\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:45:44.909217 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:44.909180    2562 generic.go:356] "Generic (PLEG): container finished" podID="dd26d755-6e39-49b6-b956-98d5709b58b0" containerID="ade2848bc4fb1509e4ba34f9270f68bafdaf92e451434a833ccb5987ee424d6e" exitCode=0
Sep 11 17:45:44.909671 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:44.909230    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq"
Sep 11 17:45:44.909671 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:44.909257    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq" event={"ID":"dd26d755-6e39-49b6-b956-98d5709b58b0","Type":"ContainerDied","Data":"ade2848bc4fb1509e4ba34f9270f68bafdaf92e451434a833ccb5987ee424d6e"}
Sep 11 17:45:44.909671 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:44.909308    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq" event={"ID":"dd26d755-6e39-49b6-b956-98d5709b58b0","Type":"ContainerDied","Data":"09e8d14b38a04b46590eaf8a48286681bde56ac37b67af7c5a799fabe075db78"}
Sep 11 17:45:44.909671 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:44.909330    2562 scope.go:117] "RemoveContainer" containerID="ade2848bc4fb1509e4ba34f9270f68bafdaf92e451434a833ccb5987ee424d6e"
Sep 11 17:45:44.919230 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:44.919210    2562 scope.go:117] "RemoveContainer" containerID="ade2848bc4fb1509e4ba34f9270f68bafdaf92e451434a833ccb5987ee424d6e"
Sep 11 17:45:44.919529 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:45:44.919506    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ade2848bc4fb1509e4ba34f9270f68bafdaf92e451434a833ccb5987ee424d6e\": container with ID starting with ade2848bc4fb1509e4ba34f9270f68bafdaf92e451434a833ccb5987ee424d6e not found: ID does not exist" containerID="ade2848bc4fb1509e4ba34f9270f68bafdaf92e451434a833ccb5987ee424d6e"
Sep 11 17:45:44.919627 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:44.919536    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ade2848bc4fb1509e4ba34f9270f68bafdaf92e451434a833ccb5987ee424d6e"} err="failed to get container status \"ade2848bc4fb1509e4ba34f9270f68bafdaf92e451434a833ccb5987ee424d6e\": rpc error: code = NotFound desc = could not find container \"ade2848bc4fb1509e4ba34f9270f68bafdaf92e451434a833ccb5987ee424d6e\": container with ID starting with ade2848bc4fb1509e4ba34f9270f68bafdaf92e451434a833ccb5987ee424d6e not found: ID does not exist"
Sep 11 17:45:44.932456 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:44.932428    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq"]
Sep 11 17:45:44.938870 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:44.938843    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-144bfa-cbf94b9fd-8lnbq"]
Sep 11 17:45:46.125318 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:45:46.125282    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd26d755-6e39-49b6-b956-98d5709b58b0" path="/var/lib/kubelet/pods/dd26d755-6e39-49b6-b956-98d5709b58b0/volumes"
Sep 11 17:46:28.286832 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:46:28.286796    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6fc9e6/auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5"]
Sep 11 17:46:43.305811 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:46:43.305740    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5" podUID="671e43ce-84d2-4f03-b15c-dc98bec4d898" containerName="main" containerID="cri-o://1b88f61bf26d44c43308ac5d3df3968065b2cff082ec02e308a73113811d6432" gracePeriod=45
Sep 11 17:46:43.563987 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:46:43.563919    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5"
Sep 11 17:46:43.660410 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:46:43.660368    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/671e43ce-84d2-4f03-b15c-dc98bec4d898-tmp-dir\") pod \"671e43ce-84d2-4f03-b15c-dc98bec4d898\" (UID: \"671e43ce-84d2-4f03-b15c-dc98bec4d898\") "
Sep 11 17:46:43.660607 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:46:43.660421    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/671e43ce-84d2-4f03-b15c-dc98bec4d898-tls-certs\") pod \"671e43ce-84d2-4f03-b15c-dc98bec4d898\" (UID: \"671e43ce-84d2-4f03-b15c-dc98bec4d898\") "
Sep 11 17:46:43.660607 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:46:43.660493    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/671e43ce-84d2-4f03-b15c-dc98bec4d898-model-cache\") pod \"671e43ce-84d2-4f03-b15c-dc98bec4d898\" (UID: \"671e43ce-84d2-4f03-b15c-dc98bec4d898\") "
Sep 11 17:46:43.660607 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:46:43.660523    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/671e43ce-84d2-4f03-b15c-dc98bec4d898-home\") pod \"671e43ce-84d2-4f03-b15c-dc98bec4d898\" (UID: \"671e43ce-84d2-4f03-b15c-dc98bec4d898\") "
Sep 11 17:46:43.660607 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:46:43.660585    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7f92\" (UniqueName: \"kubernetes.io/projected/671e43ce-84d2-4f03-b15c-dc98bec4d898-kube-api-access-l7f92\") pod \"671e43ce-84d2-4f03-b15c-dc98bec4d898\" (UID: \"671e43ce-84d2-4f03-b15c-dc98bec4d898\") "
Sep 11 17:46:43.660849 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:46:43.660620    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/671e43ce-84d2-4f03-b15c-dc98bec4d898-dshm\") pod \"671e43ce-84d2-4f03-b15c-dc98bec4d898\" (UID: \"671e43ce-84d2-4f03-b15c-dc98bec4d898\") "
Sep 11 17:46:43.660849 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:46:43.660691    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/671e43ce-84d2-4f03-b15c-dc98bec4d898-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "671e43ce-84d2-4f03-b15c-dc98bec4d898" (UID: "671e43ce-84d2-4f03-b15c-dc98bec4d898"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:46:43.660956 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:46:43.660839    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/671e43ce-84d2-4f03-b15c-dc98bec4d898-model-cache" (OuterVolumeSpecName: "model-cache") pod "671e43ce-84d2-4f03-b15c-dc98bec4d898" (UID: "671e43ce-84d2-4f03-b15c-dc98bec4d898"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:46:43.660956 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:46:43.660868    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/671e43ce-84d2-4f03-b15c-dc98bec4d898-home" (OuterVolumeSpecName: "home") pod "671e43ce-84d2-4f03-b15c-dc98bec4d898" (UID: "671e43ce-84d2-4f03-b15c-dc98bec4d898"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:46:43.660956 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:46:43.660915    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/671e43ce-84d2-4f03-b15c-dc98bec4d898-tmp-dir\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:46:43.663057 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:46:43.663024    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/671e43ce-84d2-4f03-b15c-dc98bec4d898-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "671e43ce-84d2-4f03-b15c-dc98bec4d898" (UID: "671e43ce-84d2-4f03-b15c-dc98bec4d898"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:46:43.663057 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:46:43.663053    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/671e43ce-84d2-4f03-b15c-dc98bec4d898-dshm" (OuterVolumeSpecName: "dshm") pod "671e43ce-84d2-4f03-b15c-dc98bec4d898" (UID: "671e43ce-84d2-4f03-b15c-dc98bec4d898"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:46:43.663244 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:46:43.663218    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/671e43ce-84d2-4f03-b15c-dc98bec4d898-kube-api-access-l7f92" (OuterVolumeSpecName: "kube-api-access-l7f92") pod "671e43ce-84d2-4f03-b15c-dc98bec4d898" (UID: "671e43ce-84d2-4f03-b15c-dc98bec4d898"). InnerVolumeSpecName "kube-api-access-l7f92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:46:43.761921 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:46:43.761882    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/671e43ce-84d2-4f03-b15c-dc98bec4d898-dshm\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:46:43.761921 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:46:43.761914    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/671e43ce-84d2-4f03-b15c-dc98bec4d898-tls-certs\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:46:43.761921 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:46:43.761924    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/671e43ce-84d2-4f03-b15c-dc98bec4d898-model-cache\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:46:43.761921 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:46:43.761933    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/671e43ce-84d2-4f03-b15c-dc98bec4d898-home\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:46:43.762235 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:46:43.761942    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7f92\" (UniqueName: \"kubernetes.io/projected/671e43ce-84d2-4f03-b15c-dc98bec4d898-kube-api-access-l7f92\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:46:44.203153 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:46:44.203120    2562 generic.go:356] "Generic (PLEG): container finished" podID="671e43ce-84d2-4f03-b15c-dc98bec4d898" containerID="1b88f61bf26d44c43308ac5d3df3968065b2cff082ec02e308a73113811d6432" exitCode=0
Sep 11 17:46:44.203153 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:46:44.203153    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5"
Sep 11 17:46:44.203352 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:46:44.203182    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5" event={"ID":"671e43ce-84d2-4f03-b15c-dc98bec4d898","Type":"ContainerDied","Data":"1b88f61bf26d44c43308ac5d3df3968065b2cff082ec02e308a73113811d6432"}
Sep 11 17:46:44.203352 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:46:44.203217    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6fc9e6/auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5" event={"ID":"671e43ce-84d2-4f03-b15c-dc98bec4d898","Type":"ContainerDied","Data":"44cb2b2fa64d07882bf9f3838aa44a79d9487e2ad4d8883bba6edd94b466e96e"}
Sep 11 17:46:44.203352 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:46:44.203236    2562 scope.go:117] "RemoveContainer" containerID="1b88f61bf26d44c43308ac5d3df3968065b2cff082ec02e308a73113811d6432"
Sep 11 17:46:44.211479 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:46:44.211459    2562 scope.go:117] "RemoveContainer" containerID="1b88f61bf26d44c43308ac5d3df3968065b2cff082ec02e308a73113811d6432"
Sep 11 17:46:44.211744 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:46:44.211723    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b88f61bf26d44c43308ac5d3df3968065b2cff082ec02e308a73113811d6432\": container with ID starting with 1b88f61bf26d44c43308ac5d3df3968065b2cff082ec02e308a73113811d6432 not found: ID does not exist" containerID="1b88f61bf26d44c43308ac5d3df3968065b2cff082ec02e308a73113811d6432"
Sep 11 17:46:44.211807 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:46:44.211755    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b88f61bf26d44c43308ac5d3df3968065b2cff082ec02e308a73113811d6432"} err="failed to get container status \"1b88f61bf26d44c43308ac5d3df3968065b2cff082ec02e308a73113811d6432\": rpc error: code = NotFound desc = could not find container \"1b88f61bf26d44c43308ac5d3df3968065b2cff082ec02e308a73113811d6432\": container with ID starting with 1b88f61bf26d44c43308ac5d3df3968065b2cff082ec02e308a73113811d6432 not found: ID does not exist"
Sep 11 17:46:44.219234 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:46:44.219213    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6fc9e6/auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5"]
Sep 11 17:46:44.223297 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:46:44.223277    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6fc9e6/auth-test-model-6fc9e6-kserve-77fc6c9ff-hllc5"]
Sep 11 17:46:46.125081 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:46:46.125044    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="671e43ce-84d2-4f03-b15c-dc98bec4d898" path="/var/lib/kubelet/pods/671e43ce-84d2-4f03-b15c-dc98bec4d898/volumes"
Sep 11 17:50:16.283645 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.283569    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-027edd-5f69499c7-zzcvq"]
Sep 11 17:50:16.284142 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.283915    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="671e43ce-84d2-4f03-b15c-dc98bec4d898" containerName="main"
Sep 11 17:50:16.284142 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.283928    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="671e43ce-84d2-4f03-b15c-dc98bec4d898" containerName="main"
Sep 11 17:50:16.284142 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.283938    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39cd2308-d436-41ca-b9e8-02078533374e" containerName="payload-processing"
Sep 11 17:50:16.284142 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.283944    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="39cd2308-d436-41ca-b9e8-02078533374e" containerName="payload-processing"
Sep 11 17:50:16.284142 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.283951    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcef6d19-ab56-4c36-8725-ce2253241999" containerName="main"
Sep 11 17:50:16.284142 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.283957    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcef6d19-ab56-4c36-8725-ce2253241999" containerName="main"
Sep 11 17:50:16.284142 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.283967    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00b26df4-2ff6-45f3-b020-e1854b2060d6" containerName="istio-proxy"
Sep 11 17:50:16.284142 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.283973    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="00b26df4-2ff6-45f3-b020-e1854b2060d6" containerName="istio-proxy"
Sep 11 17:50:16.284142 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.283982    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c8990a4-279f-40a5-bf70-a148874ff90f" containerName="cleanup"
Sep 11 17:50:16.284142 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.283987    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c8990a4-279f-40a5-bf70-a148874ff90f" containerName="cleanup"
Sep 11 17:50:16.284142 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.284010    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd26d755-6e39-49b6-b956-98d5709b58b0" containerName="maas-api"
Sep 11 17:50:16.284142 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.284015    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd26d755-6e39-49b6-b956-98d5709b58b0" containerName="maas-api"
Sep 11 17:50:16.284142 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.284026    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f64a7c90-1859-4bb7-afa8-76b8c51f2527" containerName="payload-pre-processing"
Sep 11 17:50:16.284142 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.284031    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f64a7c90-1859-4bb7-afa8-76b8c51f2527" containerName="payload-pre-processing"
Sep 11 17:50:16.284142 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.284081    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcef6d19-ab56-4c36-8725-ce2253241999" containerName="main"
Sep 11 17:50:16.284142 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.284089    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f64a7c90-1859-4bb7-afa8-76b8c51f2527" containerName="payload-pre-processing"
Sep 11 17:50:16.284142 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.284094    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c8990a4-279f-40a5-bf70-a148874ff90f" containerName="cleanup"
Sep 11 17:50:16.284142 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.284101    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="00b26df4-2ff6-45f3-b020-e1854b2060d6" containerName="istio-proxy"
Sep 11 17:50:16.284142 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.284106    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="39cd2308-d436-41ca-b9e8-02078533374e" containerName="payload-processing"
Sep 11 17:50:16.284142 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.284112    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="671e43ce-84d2-4f03-b15c-dc98bec4d898" containerName="main"
Sep 11 17:50:16.284142 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.284117    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd26d755-6e39-49b6-b956-98d5709b58b0" containerName="maas-api"
Sep 11 17:50:16.286981 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.286964    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-027edd-5f69499c7-zzcvq"
Sep 11 17:50:16.289488 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.289468    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-027edd-dockercfg-ktxmh\""
Sep 11 17:50:16.289785 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.289763    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-027edd\""
Sep 11 17:50:16.303373 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.303344    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-027edd-5f69499c7-zzcvq"]
Sep 11 17:50:16.324704 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.324669    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-027edd-7ddc7599db-z6cb2"]
Sep 11 17:50:16.325099 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.325084    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00b26df4-2ff6-45f3-b020-e1854b2060d6" containerName="istio-proxy"
Sep 11 17:50:16.325144 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.325102    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="00b26df4-2ff6-45f3-b020-e1854b2060d6" containerName="istio-proxy"
Sep 11 17:50:16.325186 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.325160    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="00b26df4-2ff6-45f3-b020-e1854b2060d6" containerName="istio-proxy"
Sep 11 17:50:16.328098 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.328079    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027edd-7ddc7599db-z6cb2"
Sep 11 17:50:16.338576 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.338547    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-027edd-7ddc7599db-z6cb2"]
Sep 11 17:50:16.372808 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.372743    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a52eaa4-863e-4526-8ccd-aff1a26fb1b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-027edd-7ddc7599db-z6cb2\" (UID: \"7a52eaa4-863e-4526-8ccd-aff1a26fb1b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027edd-7ddc7599db-z6cb2"
Sep 11 17:50:16.373097 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.373071    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjtbw\" (UniqueName: \"kubernetes.io/projected/7a52eaa4-863e-4526-8ccd-aff1a26fb1b4-kube-api-access-jjtbw\") pod \"payload-pre-processing-e2e-ipp-a-027edd-7ddc7599db-z6cb2\" (UID: \"7a52eaa4-863e-4526-8ccd-aff1a26fb1b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027edd-7ddc7599db-z6cb2"
Sep 11 17:50:16.373294 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.373275    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgvsz\" (UniqueName: \"kubernetes.io/projected/163e963d-b4a7-40c2-b6a9-5df5b848e182-kube-api-access-dgvsz\") pod \"payload-processing-e2e-ipp-a-027edd-5f69499c7-zzcvq\" (UID: \"163e963d-b4a7-40c2-b6a9-5df5b848e182\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-027edd-5f69499c7-zzcvq"
Sep 11 17:50:16.373463 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.373444    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/163e963d-b4a7-40c2-b6a9-5df5b848e182-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-027edd-5f69499c7-zzcvq\" (UID: \"163e963d-b4a7-40c2-b6a9-5df5b848e182\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-027edd-5f69499c7-zzcvq"
Sep 11 17:50:16.474424 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.474374    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a52eaa4-863e-4526-8ccd-aff1a26fb1b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-027edd-7ddc7599db-z6cb2\" (UID: \"7a52eaa4-863e-4526-8ccd-aff1a26fb1b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027edd-7ddc7599db-z6cb2"
Sep 11 17:50:16.474636 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.474435    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjtbw\" (UniqueName: \"kubernetes.io/projected/7a52eaa4-863e-4526-8ccd-aff1a26fb1b4-kube-api-access-jjtbw\") pod \"payload-pre-processing-e2e-ipp-a-027edd-7ddc7599db-z6cb2\" (UID: \"7a52eaa4-863e-4526-8ccd-aff1a26fb1b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027edd-7ddc7599db-z6cb2"
Sep 11 17:50:16.474636 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.474505    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgvsz\" (UniqueName: \"kubernetes.io/projected/163e963d-b4a7-40c2-b6a9-5df5b848e182-kube-api-access-dgvsz\") pod \"payload-processing-e2e-ipp-a-027edd-5f69499c7-zzcvq\" (UID: \"163e963d-b4a7-40c2-b6a9-5df5b848e182\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-027edd-5f69499c7-zzcvq"
Sep 11 17:50:16.474636 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.474559    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/163e963d-b4a7-40c2-b6a9-5df5b848e182-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-027edd-5f69499c7-zzcvq\" (UID: \"163e963d-b4a7-40c2-b6a9-5df5b848e182\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-027edd-5f69499c7-zzcvq"
Sep 11 17:50:16.475260 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.475233    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a52eaa4-863e-4526-8ccd-aff1a26fb1b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-027edd-7ddc7599db-z6cb2\" (UID: \"7a52eaa4-863e-4526-8ccd-aff1a26fb1b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027edd-7ddc7599db-z6cb2"
Sep 11 17:50:16.475515 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.475494    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/163e963d-b4a7-40c2-b6a9-5df5b848e182-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-027edd-5f69499c7-zzcvq\" (UID: \"163e963d-b4a7-40c2-b6a9-5df5b848e182\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-027edd-5f69499c7-zzcvq"
Sep 11 17:50:16.483346 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.483316    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgvsz\" (UniqueName: \"kubernetes.io/projected/163e963d-b4a7-40c2-b6a9-5df5b848e182-kube-api-access-dgvsz\") pod \"payload-processing-e2e-ipp-a-027edd-5f69499c7-zzcvq\" (UID: \"163e963d-b4a7-40c2-b6a9-5df5b848e182\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-027edd-5f69499c7-zzcvq"
Sep 11 17:50:16.483470 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.483420    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjtbw\" (UniqueName: \"kubernetes.io/projected/7a52eaa4-863e-4526-8ccd-aff1a26fb1b4-kube-api-access-jjtbw\") pod \"payload-pre-processing-e2e-ipp-a-027edd-7ddc7599db-z6cb2\" (UID: \"7a52eaa4-863e-4526-8ccd-aff1a26fb1b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027edd-7ddc7599db-z6cb2"
Sep 11 17:50:16.597518 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.597486    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-027edd-5f69499c7-zzcvq"
Sep 11 17:50:16.640605 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.640566    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027edd-7ddc7599db-z6cb2"
Sep 11 17:50:16.884419 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.884385    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-027edd-5f69499c7-zzcvq"]
Sep 11 17:50:16.885788 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:50:16.885753    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod163e963d_b4a7_40c2_b6a9_5df5b848e182.slice/crio-57f62bd02684f6b122b152da01d3b12ef21e66da9b52b399dfa95345e8154983 WatchSource:0}: Error finding container 57f62bd02684f6b122b152da01d3b12ef21e66da9b52b399dfa95345e8154983: Status 404 returned error can't find the container with id 57f62bd02684f6b122b152da01d3b12ef21e66da9b52b399dfa95345e8154983
Sep 11 17:50:16.941178 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:16.941154    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-027edd-7ddc7599db-z6cb2"]
Sep 11 17:50:16.948224 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:50:16.948201    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a52eaa4_863e_4526_8ccd_aff1a26fb1b4.slice/crio-5442208f2433f0bc24df1f9312626c58b1aa8fba6e58088be36635aec1eb89bb WatchSource:0}: Error finding container 5442208f2433f0bc24df1f9312626c58b1aa8fba6e58088be36635aec1eb89bb: Status 404 returned error can't find the container with id 5442208f2433f0bc24df1f9312626c58b1aa8fba6e58088be36635aec1eb89bb
Sep 11 17:50:17.199719 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:17.199624    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027edd-7ddc7599db-z6cb2" event={"ID":"7a52eaa4-863e-4526-8ccd-aff1a26fb1b4","Type":"ContainerStarted","Data":"8ca7e221a01940d5bf407a187769540906bc64eba479ac062a6878777c3a1d6a"}
Sep 11 17:50:17.199719 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:17.199704    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027edd-7ddc7599db-z6cb2" event={"ID":"7a52eaa4-863e-4526-8ccd-aff1a26fb1b4","Type":"ContainerStarted","Data":"5442208f2433f0bc24df1f9312626c58b1aa8fba6e58088be36635aec1eb89bb"}
Sep 11 17:50:17.199956 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:17.199917    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027edd-7ddc7599db-z6cb2"
Sep 11 17:50:17.201612 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:17.201580    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-027edd-5f69499c7-zzcvq" event={"ID":"163e963d-b4a7-40c2-b6a9-5df5b848e182","Type":"ContainerStarted","Data":"813779478abe0fbefa9d7b0d3a255a2fcf407c61c1f209e284e0dfdcb81f82a2"}
Sep 11 17:50:17.201792 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:17.201635    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-027edd-5f69499c7-zzcvq" event={"ID":"163e963d-b4a7-40c2-b6a9-5df5b848e182","Type":"ContainerStarted","Data":"57f62bd02684f6b122b152da01d3b12ef21e66da9b52b399dfa95345e8154983"}
Sep 11 17:50:17.201792 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:17.201665    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-027edd-5f69499c7-zzcvq"
Sep 11 17:50:17.223018 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:17.222939    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027edd-7ddc7599db-z6cb2" podStartSLOduration=1.222921757 podStartE2EDuration="1.222921757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:50:17.220122659 +0000 UTC m=+1897.614040115" watchObservedRunningTime="2026-09-11 17:50:17.222921757 +0000 UTC m=+1897.616839213"
Sep 11 17:50:17.247178 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:17.247116    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-027edd-5f69499c7-zzcvq" podStartSLOduration=1.247097114 podStartE2EDuration="1.247097114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:50:17.245194798 +0000 UTC m=+1897.639112255" watchObservedRunningTime="2026-09-11 17:50:17.247097114 +0000 UTC m=+1897.641014570"
Sep 11 17:50:28.209013 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:28.208965    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-027edd-5f69499c7-zzcvq"
Sep 11 17:50:28.209496 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:50:28.209388    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027edd-7ddc7599db-z6cb2"
Sep 11 17:51:12.873084 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:12.873056    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-lr9nq_5f6b61e9-0864-47bb-aeec-df01bbf50d34/payload-processing/0.log"
Sep 11 17:51:43.696106 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:43.696027    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-027edd-5f69499c7-zzcvq_163e963d-b4a7-40c2-b6a9-5df5b848e182/payload-processing/0.log"
Sep 11 17:51:43.808918 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:43.808887    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-lr9nq_5f6b61e9-0864-47bb-aeec-df01bbf50d34/payload-processing/0.log"
Sep 11 17:51:44.519487 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:44.519449    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"]
Sep 11 17:51:44.522678 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:44.522658    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"
Sep 11 17:51:44.530729 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:44.530705    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-574f8b-openshift-default-dockercfg-7skl8\""
Sep 11 17:51:44.544442 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:44.544420    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"]
Sep 11 17:51:44.616197 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:44.616154    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2da9345-6535-464d-9048-0e6008496838-workload-socket\") pod \"e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g\" (UID: \"f2da9345-6535-464d-9048-0e6008496838\") " pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"
Sep 11 17:51:44.616399 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:44.616237    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2da9345-6535-464d-9048-0e6008496838-istio-envoy\") pod \"e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g\" (UID: \"f2da9345-6535-464d-9048-0e6008496838\") " pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"
Sep 11 17:51:44.616399 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:44.616285    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2da9345-6535-464d-9048-0e6008496838-credential-socket\") pod \"e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g\" (UID: \"f2da9345-6535-464d-9048-0e6008496838\") " pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"
Sep 11 17:51:44.616399 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:44.616321    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2da9345-6535-464d-9048-0e6008496838-istio-podinfo\") pod \"e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g\" (UID: \"f2da9345-6535-464d-9048-0e6008496838\") " pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"
Sep 11 17:51:44.616399 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:44.616370    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgzqw\" (UniqueName: \"kubernetes.io/projected/f2da9345-6535-464d-9048-0e6008496838-kube-api-access-sgzqw\") pod \"e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g\" (UID: \"f2da9345-6535-464d-9048-0e6008496838\") " pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"
Sep 11 17:51:44.616648 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:44.616398    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2da9345-6535-464d-9048-0e6008496838-istio-token\") pod \"e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g\" (UID: \"f2da9345-6535-464d-9048-0e6008496838\") " pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"
Sep 11 17:51:44.616648 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:44.616429    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2da9345-6535-464d-9048-0e6008496838-workload-certs\") pod \"e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g\" (UID: \"f2da9345-6535-464d-9048-0e6008496838\") " pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"
Sep 11 17:51:44.616648 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:44.616498    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2da9345-6535-464d-9048-0e6008496838-istio-data\") pod \"e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g\" (UID: \"f2da9345-6535-464d-9048-0e6008496838\") " pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"
Sep 11 17:51:44.616648 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:44.616535    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2da9345-6535-464d-9048-0e6008496838-istiod-ca-cert\") pod \"e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g\" (UID: \"f2da9345-6535-464d-9048-0e6008496838\") " pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"
Sep 11 17:51:44.718023 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:44.717960    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2da9345-6535-464d-9048-0e6008496838-workload-socket\") pod \"e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g\" (UID: \"f2da9345-6535-464d-9048-0e6008496838\") " pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"
Sep 11 17:51:44.718524 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:44.718058    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2da9345-6535-464d-9048-0e6008496838-istio-envoy\") pod \"e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g\" (UID: \"f2da9345-6535-464d-9048-0e6008496838\") " pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"
Sep 11 17:51:44.718524 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:44.718098    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2da9345-6535-464d-9048-0e6008496838-credential-socket\") pod \"e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g\" (UID: \"f2da9345-6535-464d-9048-0e6008496838\") " pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"
Sep 11 17:51:44.718524 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:44.718129    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2da9345-6535-464d-9048-0e6008496838-istio-podinfo\") pod \"e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g\" (UID: \"f2da9345-6535-464d-9048-0e6008496838\") " pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"
Sep 11 17:51:44.718524 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:44.718182    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgzqw\" (UniqueName: \"kubernetes.io/projected/f2da9345-6535-464d-9048-0e6008496838-kube-api-access-sgzqw\") pod \"e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g\" (UID: \"f2da9345-6535-464d-9048-0e6008496838\") " pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"
Sep 11 17:51:44.718524 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:44.718209    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2da9345-6535-464d-9048-0e6008496838-istio-token\") pod \"e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g\" (UID: \"f2da9345-6535-464d-9048-0e6008496838\") " pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"
Sep 11 17:51:44.718524 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:44.718238    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2da9345-6535-464d-9048-0e6008496838-workload-certs\") pod \"e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g\" (UID: \"f2da9345-6535-464d-9048-0e6008496838\") " pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"
Sep 11 17:51:44.718524 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:44.718272    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2da9345-6535-464d-9048-0e6008496838-istio-data\") pod \"e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g\" (UID: \"f2da9345-6535-464d-9048-0e6008496838\") " pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"
Sep 11 17:51:44.718524 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:44.718316    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2da9345-6535-464d-9048-0e6008496838-istiod-ca-cert\") pod \"e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g\" (UID: \"f2da9345-6535-464d-9048-0e6008496838\") " pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"
Sep 11 17:51:44.718524 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:44.718413    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2da9345-6535-464d-9048-0e6008496838-workload-socket\") pod \"e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g\" (UID: \"f2da9345-6535-464d-9048-0e6008496838\") " pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"
Sep 11 17:51:44.719066 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:44.718722    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2da9345-6535-464d-9048-0e6008496838-credential-socket\") pod \"e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g\" (UID: \"f2da9345-6535-464d-9048-0e6008496838\") " pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"
Sep 11 17:51:44.719066 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:44.718858    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2da9345-6535-464d-9048-0e6008496838-workload-certs\") pod \"e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g\" (UID: \"f2da9345-6535-464d-9048-0e6008496838\") " pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"
Sep 11 17:51:44.719066 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:44.718942    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2da9345-6535-464d-9048-0e6008496838-istio-data\") pod \"e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g\" (UID: \"f2da9345-6535-464d-9048-0e6008496838\") " pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"
Sep 11 17:51:44.719222 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:44.719153    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2da9345-6535-464d-9048-0e6008496838-istiod-ca-cert\") pod \"e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g\" (UID: \"f2da9345-6535-464d-9048-0e6008496838\") " pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"
Sep 11 17:51:44.720569 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:44.720543    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2da9345-6535-464d-9048-0e6008496838-istio-envoy\") pod \"e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g\" (UID: \"f2da9345-6535-464d-9048-0e6008496838\") " pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"
Sep 11 17:51:44.720844 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:44.720829    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2da9345-6535-464d-9048-0e6008496838-istio-podinfo\") pod \"e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g\" (UID: \"f2da9345-6535-464d-9048-0e6008496838\") " pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"
Sep 11 17:51:44.726425 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:44.726401    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgzqw\" (UniqueName: \"kubernetes.io/projected/f2da9345-6535-464d-9048-0e6008496838-kube-api-access-sgzqw\") pod \"e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g\" (UID: \"f2da9345-6535-464d-9048-0e6008496838\") " pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"
Sep 11 17:51:44.727304 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:44.727276    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2da9345-6535-464d-9048-0e6008496838-istio-token\") pod \"e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g\" (UID: \"f2da9345-6535-464d-9048-0e6008496838\") " pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"
Sep 11 17:51:44.832530 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:44.832444    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"
Sep 11 17:51:45.120482 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:45.120409    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"]
Sep 11 17:51:45.122892 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:51:45.122862    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2da9345_6535_464d_9048_0e6008496838.slice/crio-3d7daab924f79444ee0668f25abce21e8a43b32557619244238b102b5ab71f62 WatchSource:0}: Error finding container 3d7daab924f79444ee0668f25abce21e8a43b32557619244238b102b5ab71f62: Status 404 returned error can't find the container with id 3d7daab924f79444ee0668f25abce21e8a43b32557619244238b102b5ab71f62
Sep 11 17:51:45.125298 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:45.125264    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:51:45.125395 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:45.125330    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:51:45.125395 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:45.125359    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:51:45.621280 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:45.621244    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g" event={"ID":"f2da9345-6535-464d-9048-0e6008496838","Type":"ContainerStarted","Data":"d8097ed540bb4967fcec94ebb5fda35c56034ba827a8fe204197c6011ce71275"}
Sep 11 17:51:45.621447 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:45.621297    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g" event={"ID":"f2da9345-6535-464d-9048-0e6008496838","Type":"ContainerStarted","Data":"3d7daab924f79444ee0668f25abce21e8a43b32557619244238b102b5ab71f62"}
Sep 11 17:51:45.647835 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:45.647780    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g" podStartSLOduration=1.647759648 podStartE2EDuration="1.647759648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:51:45.644479728 +0000 UTC m=+1986.038397182" watchObservedRunningTime="2026-09-11 17:51:45.647759648 +0000 UTC m=+1986.041677104"
Sep 11 17:51:45.832851 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:45.832821    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"
Sep 11 17:51:46.839199 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:46.839172    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"
Sep 11 17:51:47.633189 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:47.633090    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"
Sep 11 17:51:47.640535 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:51:47.640507    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"
Sep 11 17:52:07.730883 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:07.730792    2562 generic.go:356] "Generic (PLEG): container finished" podID="f2da9345-6535-464d-9048-0e6008496838" containerID="d8097ed540bb4967fcec94ebb5fda35c56034ba827a8fe204197c6011ce71275" exitCode=0
Sep 11 17:52:07.730883 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:07.730865    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g" event={"ID":"f2da9345-6535-464d-9048-0e6008496838","Type":"ContainerDied","Data":"d8097ed540bb4967fcec94ebb5fda35c56034ba827a8fe204197c6011ce71275"}
Sep 11 17:52:07.731363 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:07.731322    2562 scope.go:117] "RemoveContainer" containerID="d8097ed540bb4967fcec94ebb5fda35c56034ba827a8fe204197c6011ce71275"
Sep 11 17:52:07.732393 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:07.732361    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:52:07.732562 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:07.732445    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:52:07.732562 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:07.732494    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:52:08.738717 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:08.738683    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g" event={"ID":"f2da9345-6535-464d-9048-0e6008496838","Type":"ContainerStarted","Data":"3803e948d79d1dc7f655adec4bdf64f3fbc8f551ab99c30d705a4b1a6b1f521e"}
Sep 11 17:52:08.833035 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:08.832987    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"
Sep 11 17:52:08.833249 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:08.833226    2562 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 11 17:52:08.833337 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:08.833319    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g" podUID="f2da9345-6535-464d-9048-0e6008496838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 11 17:52:09.833517 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:09.833478    2562 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 11 17:52:09.834051 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:09.833541    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g" podUID="f2da9345-6535-464d-9048-0e6008496838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 11 17:52:10.832881 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:10.832844    2562 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 11 17:52:10.833095 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:10.832901    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g" podUID="f2da9345-6535-464d-9048-0e6008496838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 11 17:52:11.833422 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:11.833384    2562 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 11 17:52:11.833905 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:11.833443    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g" podUID="f2da9345-6535-464d-9048-0e6008496838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 11 17:52:12.833366 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:12.833333    2562 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 11 17:52:12.833539 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:12.833387    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g" podUID="f2da9345-6535-464d-9048-0e6008496838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 11 17:52:13.833556 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:13.833519    2562 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 11 17:52:13.833986 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:13.833587    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g" podUID="f2da9345-6535-464d-9048-0e6008496838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 11 17:52:14.837444 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:14.837413    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"
Sep 11 17:52:14.837862 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:14.837674    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"
Sep 11 17:52:14.838373 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:14.838356    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"
Sep 11 17:52:36.521605 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:36.521573    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"]
Sep 11 17:52:36.522150 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:36.521793    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g" podUID="f2da9345-6535-464d-9048-0e6008496838" containerName="istio-proxy" containerID="cri-o://3803e948d79d1dc7f655adec4bdf64f3fbc8f551ab99c30d705a4b1a6b1f521e" gracePeriod=30
Sep 11 17:52:41.782830 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.782803    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"
Sep 11 17:52:41.875679 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.875646    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2da9345-6535-464d-9048-0e6008496838-credential-socket\") pod \"f2da9345-6535-464d-9048-0e6008496838\" (UID: \"f2da9345-6535-464d-9048-0e6008496838\") "
Sep 11 17:52:41.875679 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.875685    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2da9345-6535-464d-9048-0e6008496838-istiod-ca-cert\") pod \"f2da9345-6535-464d-9048-0e6008496838\" (UID: \"f2da9345-6535-464d-9048-0e6008496838\") "
Sep 11 17:52:41.875930 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.875719    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2da9345-6535-464d-9048-0e6008496838-istio-envoy\") pod \"f2da9345-6535-464d-9048-0e6008496838\" (UID: \"f2da9345-6535-464d-9048-0e6008496838\") "
Sep 11 17:52:41.875930 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.875772    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2da9345-6535-464d-9048-0e6008496838-istio-data\") pod \"f2da9345-6535-464d-9048-0e6008496838\" (UID: \"f2da9345-6535-464d-9048-0e6008496838\") "
Sep 11 17:52:41.875930 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.875801    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgzqw\" (UniqueName: \"kubernetes.io/projected/f2da9345-6535-464d-9048-0e6008496838-kube-api-access-sgzqw\") pod \"f2da9345-6535-464d-9048-0e6008496838\" (UID: \"f2da9345-6535-464d-9048-0e6008496838\") "
Sep 11 17:52:41.875930 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.875868    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2da9345-6535-464d-9048-0e6008496838-istio-podinfo\") pod \"f2da9345-6535-464d-9048-0e6008496838\" (UID: \"f2da9345-6535-464d-9048-0e6008496838\") "
Sep 11 17:52:41.875930 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.875890    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2da9345-6535-464d-9048-0e6008496838-istio-token\") pod \"f2da9345-6535-464d-9048-0e6008496838\" (UID: \"f2da9345-6535-464d-9048-0e6008496838\") "
Sep 11 17:52:41.876243 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.875943    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2da9345-6535-464d-9048-0e6008496838-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f2da9345-6535-464d-9048-0e6008496838" (UID: "f2da9345-6535-464d-9048-0e6008496838"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:52:41.876243 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.876027    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2da9345-6535-464d-9048-0e6008496838-workload-socket\") pod \"f2da9345-6535-464d-9048-0e6008496838\" (UID: \"f2da9345-6535-464d-9048-0e6008496838\") "
Sep 11 17:52:41.876243 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.876103    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2da9345-6535-464d-9048-0e6008496838-workload-certs\") pod \"f2da9345-6535-464d-9048-0e6008496838\" (UID: \"f2da9345-6535-464d-9048-0e6008496838\") "
Sep 11 17:52:41.876243 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.876103    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2da9345-6535-464d-9048-0e6008496838-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f2da9345-6535-464d-9048-0e6008496838" (UID: "f2da9345-6535-464d-9048-0e6008496838"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:52:41.876243 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.876130    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2da9345-6535-464d-9048-0e6008496838-istio-data" (OuterVolumeSpecName: "istio-data") pod "f2da9345-6535-464d-9048-0e6008496838" (UID: "f2da9345-6535-464d-9048-0e6008496838"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:52:41.876503 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.876279    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2da9345-6535-464d-9048-0e6008496838-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f2da9345-6535-464d-9048-0e6008496838" (UID: "f2da9345-6535-464d-9048-0e6008496838"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:52:41.876503 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.876408    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2da9345-6535-464d-9048-0e6008496838-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f2da9345-6535-464d-9048-0e6008496838" (UID: "f2da9345-6535-464d-9048-0e6008496838"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:52:41.876615 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.876547    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2da9345-6535-464d-9048-0e6008496838-credential-socket\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:52:41.876615 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.876568    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2da9345-6535-464d-9048-0e6008496838-istiod-ca-cert\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:52:41.876615 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.876583    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2da9345-6535-464d-9048-0e6008496838-istio-data\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:52:41.876615 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.876596    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2da9345-6535-464d-9048-0e6008496838-workload-socket\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:52:41.876615 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.876610    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2da9345-6535-464d-9048-0e6008496838-workload-certs\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:52:41.878132 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.878095    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2da9345-6535-464d-9048-0e6008496838-kube-api-access-sgzqw" (OuterVolumeSpecName: "kube-api-access-sgzqw") pod "f2da9345-6535-464d-9048-0e6008496838" (UID: "f2da9345-6535-464d-9048-0e6008496838"). InnerVolumeSpecName "kube-api-access-sgzqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:52:41.878132 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.878106    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2da9345-6535-464d-9048-0e6008496838-istio-token" (OuterVolumeSpecName: "istio-token") pod "f2da9345-6535-464d-9048-0e6008496838" (UID: "f2da9345-6535-464d-9048-0e6008496838"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:52:41.878280 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.878151    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2da9345-6535-464d-9048-0e6008496838-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f2da9345-6535-464d-9048-0e6008496838" (UID: "f2da9345-6535-464d-9048-0e6008496838"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 17:52:41.878280 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.878190    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2da9345-6535-464d-9048-0e6008496838-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f2da9345-6535-464d-9048-0e6008496838" (UID: "f2da9345-6535-464d-9048-0e6008496838"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:52:41.913571 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.913539    2562 generic.go:356] "Generic (PLEG): container finished" podID="f2da9345-6535-464d-9048-0e6008496838" containerID="3803e948d79d1dc7f655adec4bdf64f3fbc8f551ab99c30d705a4b1a6b1f521e" exitCode=0
Sep 11 17:52:41.913711 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.913595    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"
Sep 11 17:52:41.913711 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.913626    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g" event={"ID":"f2da9345-6535-464d-9048-0e6008496838","Type":"ContainerDied","Data":"3803e948d79d1dc7f655adec4bdf64f3fbc8f551ab99c30d705a4b1a6b1f521e"}
Sep 11 17:52:41.913711 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.913691    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g" event={"ID":"f2da9345-6535-464d-9048-0e6008496838","Type":"ContainerDied","Data":"3d7daab924f79444ee0668f25abce21e8a43b32557619244238b102b5ab71f62"}
Sep 11 17:52:41.913834 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.913723    2562 scope.go:117] "RemoveContainer" containerID="3803e948d79d1dc7f655adec4bdf64f3fbc8f551ab99c30d705a4b1a6b1f521e"
Sep 11 17:52:41.923110 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.923093    2562 scope.go:117] "RemoveContainer" containerID="d8097ed540bb4967fcec94ebb5fda35c56034ba827a8fe204197c6011ce71275"
Sep 11 17:52:41.931452 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.931435    2562 scope.go:117] "RemoveContainer" containerID="3803e948d79d1dc7f655adec4bdf64f3fbc8f551ab99c30d705a4b1a6b1f521e"
Sep 11 17:52:41.931727 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:52:41.931707    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3803e948d79d1dc7f655adec4bdf64f3fbc8f551ab99c30d705a4b1a6b1f521e\": container with ID starting with 3803e948d79d1dc7f655adec4bdf64f3fbc8f551ab99c30d705a4b1a6b1f521e not found: ID does not exist" containerID="3803e948d79d1dc7f655adec4bdf64f3fbc8f551ab99c30d705a4b1a6b1f521e"
Sep 11 17:52:41.931784 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.931735    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3803e948d79d1dc7f655adec4bdf64f3fbc8f551ab99c30d705a4b1a6b1f521e"} err="failed to get container status \"3803e948d79d1dc7f655adec4bdf64f3fbc8f551ab99c30d705a4b1a6b1f521e\": rpc error: code = NotFound desc = could not find container \"3803e948d79d1dc7f655adec4bdf64f3fbc8f551ab99c30d705a4b1a6b1f521e\": container with ID starting with 3803e948d79d1dc7f655adec4bdf64f3fbc8f551ab99c30d705a4b1a6b1f521e not found: ID does not exist"
Sep 11 17:52:41.931784 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.931753    2562 scope.go:117] "RemoveContainer" containerID="d8097ed540bb4967fcec94ebb5fda35c56034ba827a8fe204197c6011ce71275"
Sep 11 17:52:41.932034 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:52:41.931985    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8097ed540bb4967fcec94ebb5fda35c56034ba827a8fe204197c6011ce71275\": container with ID starting with d8097ed540bb4967fcec94ebb5fda35c56034ba827a8fe204197c6011ce71275 not found: ID does not exist" containerID="d8097ed540bb4967fcec94ebb5fda35c56034ba827a8fe204197c6011ce71275"
Sep 11 17:52:41.932102 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.932045    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8097ed540bb4967fcec94ebb5fda35c56034ba827a8fe204197c6011ce71275"} err="failed to get container status \"d8097ed540bb4967fcec94ebb5fda35c56034ba827a8fe204197c6011ce71275\": rpc error: code = NotFound desc = could not find container \"d8097ed540bb4967fcec94ebb5fda35c56034ba827a8fe204197c6011ce71275\": container with ID starting with d8097ed540bb4967fcec94ebb5fda35c56034ba827a8fe204197c6011ce71275 not found: ID does not exist"
Sep 11 17:52:41.935146 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.935125    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"]
Sep 11 17:52:41.939681 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.939662    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-574f8b-openshift-default-7d675df8d8-4pv6g"]
Sep 11 17:52:41.977904 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.977874    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgzqw\" (UniqueName: \"kubernetes.io/projected/f2da9345-6535-464d-9048-0e6008496838-kube-api-access-sgzqw\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:52:41.977904 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.977902    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2da9345-6535-464d-9048-0e6008496838-istio-podinfo\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:52:41.978099 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.977913    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2da9345-6535-464d-9048-0e6008496838-istio-token\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:52:41.978099 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:41.977922    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2da9345-6535-464d-9048-0e6008496838-istio-envoy\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:52:42.126747 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:42.126668    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2da9345-6535-464d-9048-0e6008496838" path="/var/lib/kubelet/pods/f2da9345-6535-464d-9048-0e6008496838/volumes"
Sep 11 17:52:55.766489 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:55.766454    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-027edd-5f69499c7-zzcvq"]
Sep 11 17:52:55.766977 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:55.766695    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-027edd-5f69499c7-zzcvq" podUID="163e963d-b4a7-40c2-b6a9-5df5b848e182" containerName="payload-processing" containerID="cri-o://813779478abe0fbefa9d7b0d3a255a2fcf407c61c1f209e284e0dfdcb81f82a2" gracePeriod=30
Sep 11 17:52:55.778927 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:55.776132    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-027edd-7ddc7599db-z6cb2"]
Sep 11 17:52:55.778927 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:55.776439    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027edd-7ddc7599db-z6cb2" podUID="7a52eaa4-863e-4526-8ccd-aff1a26fb1b4" containerName="payload-pre-processing" containerID="cri-o://8ca7e221a01940d5bf407a187769540906bc64eba479ac062a6878777c3a1d6a" gracePeriod=30
Sep 11 17:52:55.992977 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:55.992941    2562 generic.go:356] "Generic (PLEG): container finished" podID="163e963d-b4a7-40c2-b6a9-5df5b848e182" containerID="813779478abe0fbefa9d7b0d3a255a2fcf407c61c1f209e284e0dfdcb81f82a2" exitCode=0
Sep 11 17:52:55.993142 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:55.993035    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-027edd-5f69499c7-zzcvq" event={"ID":"163e963d-b4a7-40c2-b6a9-5df5b848e182","Type":"ContainerDied","Data":"813779478abe0fbefa9d7b0d3a255a2fcf407c61c1f209e284e0dfdcb81f82a2"}
Sep 11 17:52:55.997058 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:55.996829    2562 generic.go:356] "Generic (PLEG): container finished" podID="7a52eaa4-863e-4526-8ccd-aff1a26fb1b4" containerID="8ca7e221a01940d5bf407a187769540906bc64eba479ac062a6878777c3a1d6a" exitCode=0
Sep 11 17:52:55.997058 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:55.996951    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027edd-7ddc7599db-z6cb2" event={"ID":"7a52eaa4-863e-4526-8ccd-aff1a26fb1b4","Type":"ContainerDied","Data":"8ca7e221a01940d5bf407a187769540906bc64eba479ac062a6878777c3a1d6a"}
Sep 11 17:52:56.090699 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:56.090672    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-027edd-5f69499c7-zzcvq"
Sep 11 17:52:56.093732 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:56.093708    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027edd-7ddc7599db-z6cb2"
Sep 11 17:52:56.210145 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:56.210105    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a52eaa4-863e-4526-8ccd-aff1a26fb1b4-plugins-config-volume\") pod \"7a52eaa4-863e-4526-8ccd-aff1a26fb1b4\" (UID: \"7a52eaa4-863e-4526-8ccd-aff1a26fb1b4\") "
Sep 11 17:52:56.210145 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:56.210153    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgvsz\" (UniqueName: \"kubernetes.io/projected/163e963d-b4a7-40c2-b6a9-5df5b848e182-kube-api-access-dgvsz\") pod \"163e963d-b4a7-40c2-b6a9-5df5b848e182\" (UID: \"163e963d-b4a7-40c2-b6a9-5df5b848e182\") "
Sep 11 17:52:56.210378 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:56.210217    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/163e963d-b4a7-40c2-b6a9-5df5b848e182-plugins-config-volume\") pod \"163e963d-b4a7-40c2-b6a9-5df5b848e182\" (UID: \"163e963d-b4a7-40c2-b6a9-5df5b848e182\") "
Sep 11 17:52:56.210378 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:56.210246    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjtbw\" (UniqueName: \"kubernetes.io/projected/7a52eaa4-863e-4526-8ccd-aff1a26fb1b4-kube-api-access-jjtbw\") pod \"7a52eaa4-863e-4526-8ccd-aff1a26fb1b4\" (UID: \"7a52eaa4-863e-4526-8ccd-aff1a26fb1b4\") "
Sep 11 17:52:56.210558 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:56.210526    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a52eaa4-863e-4526-8ccd-aff1a26fb1b4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7a52eaa4-863e-4526-8ccd-aff1a26fb1b4" (UID: "7a52eaa4-863e-4526-8ccd-aff1a26fb1b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:52:56.210632 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:56.210611    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/163e963d-b4a7-40c2-b6a9-5df5b848e182-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "163e963d-b4a7-40c2-b6a9-5df5b848e182" (UID: "163e963d-b4a7-40c2-b6a9-5df5b848e182"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:52:56.212354 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:56.212326    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/163e963d-b4a7-40c2-b6a9-5df5b848e182-kube-api-access-dgvsz" (OuterVolumeSpecName: "kube-api-access-dgvsz") pod "163e963d-b4a7-40c2-b6a9-5df5b848e182" (UID: "163e963d-b4a7-40c2-b6a9-5df5b848e182"). InnerVolumeSpecName "kube-api-access-dgvsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:52:56.212354 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:56.212341    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a52eaa4-863e-4526-8ccd-aff1a26fb1b4-kube-api-access-jjtbw" (OuterVolumeSpecName: "kube-api-access-jjtbw") pod "7a52eaa4-863e-4526-8ccd-aff1a26fb1b4" (UID: "7a52eaa4-863e-4526-8ccd-aff1a26fb1b4"). InnerVolumeSpecName "kube-api-access-jjtbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:52:56.311722 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:56.311690    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjtbw\" (UniqueName: \"kubernetes.io/projected/7a52eaa4-863e-4526-8ccd-aff1a26fb1b4-kube-api-access-jjtbw\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:52:56.311722 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:56.311715    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a52eaa4-863e-4526-8ccd-aff1a26fb1b4-plugins-config-volume\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:52:56.311722 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:56.311726    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgvsz\" (UniqueName: \"kubernetes.io/projected/163e963d-b4a7-40c2-b6a9-5df5b848e182-kube-api-access-dgvsz\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:52:56.312106 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:56.311736    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/163e963d-b4a7-40c2-b6a9-5df5b848e182-plugins-config-volume\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:52:57.002920 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:57.002889    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-027edd-5f69499c7-zzcvq"
Sep 11 17:52:57.003437 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:57.002920    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-027edd-5f69499c7-zzcvq" event={"ID":"163e963d-b4a7-40c2-b6a9-5df5b848e182","Type":"ContainerDied","Data":"57f62bd02684f6b122b152da01d3b12ef21e66da9b52b399dfa95345e8154983"}
Sep 11 17:52:57.003437 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:57.002986    2562 scope.go:117] "RemoveContainer" containerID="813779478abe0fbefa9d7b0d3a255a2fcf407c61c1f209e284e0dfdcb81f82a2"
Sep 11 17:52:57.004738 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:57.004469    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027edd-7ddc7599db-z6cb2"
Sep 11 17:52:57.004738 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:57.004469    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027edd-7ddc7599db-z6cb2" event={"ID":"7a52eaa4-863e-4526-8ccd-aff1a26fb1b4","Type":"ContainerDied","Data":"5442208f2433f0bc24df1f9312626c58b1aa8fba6e58088be36635aec1eb89bb"}
Sep 11 17:52:57.012458 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:57.012443    2562 scope.go:117] "RemoveContainer" containerID="8ca7e221a01940d5bf407a187769540906bc64eba479ac062a6878777c3a1d6a"
Sep 11 17:52:57.029657 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:57.029625    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-027edd-5f69499c7-zzcvq"]
Sep 11 17:52:57.034060 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:57.034034    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-027edd-5f69499c7-zzcvq"]
Sep 11 17:52:57.043506 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:57.043481    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-027edd-7ddc7599db-z6cb2"]
Sep 11 17:52:57.047957 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:57.047936    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-027edd-7ddc7599db-z6cb2"]
Sep 11 17:52:58.126117 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:58.126085    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="163e963d-b4a7-40c2-b6a9-5df5b848e182" path="/var/lib/kubelet/pods/163e963d-b4a7-40c2-b6a9-5df5b848e182/volumes"
Sep 11 17:52:58.126569 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:52:58.126460    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a52eaa4-863e-4526-8ccd-aff1a26fb1b4" path="/var/lib/kubelet/pods/7a52eaa4-863e-4526-8ccd-aff1a26fb1b4/volumes"
Sep 11 17:54:19.472274 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:19.472236    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fac54-29819145-zjh8f"]
Sep 11 17:54:19.473898 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:19.473867    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2fac54-29819145-zjh8f"]
Sep 11 17:54:19.513128 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:19.513094    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fac54-7dfb59d654-ntv7d"]
Sep 11 17:54:19.513415 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:19.513351    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fac54-7dfb59d654-ntv7d" podUID="84c23adc-e901-4cc9-80d1-18e632314638" containerName="payload-pre-processing" containerID="cri-o://cba98995bebc03fe0d20990aad09c959be4be2f2a08caa74a423eff7d7cd4820" gracePeriod=30
Sep 11 17:54:19.774546 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:19.774512    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fac54-7dfb59d654-ntv7d"
Sep 11 17:54:19.844318 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:19.844281    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ctvm\" (UniqueName: \"kubernetes.io/projected/84c23adc-e901-4cc9-80d1-18e632314638-kube-api-access-8ctvm\") pod \"84c23adc-e901-4cc9-80d1-18e632314638\" (UID: \"84c23adc-e901-4cc9-80d1-18e632314638\") "
Sep 11 17:54:19.844522 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:19.844347    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c23adc-e901-4cc9-80d1-18e632314638-plugins-config-volume\") pod \"84c23adc-e901-4cc9-80d1-18e632314638\" (UID: \"84c23adc-e901-4cc9-80d1-18e632314638\") "
Sep 11 17:54:19.845060 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:19.844979    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84c23adc-e901-4cc9-80d1-18e632314638-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "84c23adc-e901-4cc9-80d1-18e632314638" (UID: "84c23adc-e901-4cc9-80d1-18e632314638"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:54:19.847395 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:19.847349    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c23adc-e901-4cc9-80d1-18e632314638-kube-api-access-8ctvm" (OuterVolumeSpecName: "kube-api-access-8ctvm") pod "84c23adc-e901-4cc9-80d1-18e632314638" (UID: "84c23adc-e901-4cc9-80d1-18e632314638"). InnerVolumeSpecName "kube-api-access-8ctvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:54:19.946013 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:19.945961    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ctvm\" (UniqueName: \"kubernetes.io/projected/84c23adc-e901-4cc9-80d1-18e632314638-kube-api-access-8ctvm\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:54:19.946013 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:19.946005    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c23adc-e901-4cc9-80d1-18e632314638-plugins-config-volume\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:54:20.124738 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:20.124705    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c8990a4-279f-40a5-bf70-a148874ff90f" path="/var/lib/kubelet/pods/4c8990a4-279f-40a5-bf70-a148874ff90f/volumes"
Sep 11 17:54:20.401671 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:20.401583    2562 generic.go:356] "Generic (PLEG): container finished" podID="84c23adc-e901-4cc9-80d1-18e632314638" containerID="cba98995bebc03fe0d20990aad09c959be4be2f2a08caa74a423eff7d7cd4820" exitCode=0
Sep 11 17:54:20.401671 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:20.401635    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fac54-7dfb59d654-ntv7d"
Sep 11 17:54:20.401876 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:20.401640    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fac54-7dfb59d654-ntv7d" event={"ID":"84c23adc-e901-4cc9-80d1-18e632314638","Type":"ContainerDied","Data":"cba98995bebc03fe0d20990aad09c959be4be2f2a08caa74a423eff7d7cd4820"}
Sep 11 17:54:20.401876 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:20.401785    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fac54-7dfb59d654-ntv7d" event={"ID":"84c23adc-e901-4cc9-80d1-18e632314638","Type":"ContainerDied","Data":"a6fadad520915b4bb1c997d3ef6bbcc2c803bad987a0cccd8df70d969c9575be"}
Sep 11 17:54:20.401876 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:20.401819    2562 scope.go:117] "RemoveContainer" containerID="cba98995bebc03fe0d20990aad09c959be4be2f2a08caa74a423eff7d7cd4820"
Sep 11 17:54:20.410163 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:20.410138    2562 scope.go:117] "RemoveContainer" containerID="cba98995bebc03fe0d20990aad09c959be4be2f2a08caa74a423eff7d7cd4820"
Sep 11 17:54:20.410444 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:54:20.410426    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cba98995bebc03fe0d20990aad09c959be4be2f2a08caa74a423eff7d7cd4820\": container with ID starting with cba98995bebc03fe0d20990aad09c959be4be2f2a08caa74a423eff7d7cd4820 not found: ID does not exist" containerID="cba98995bebc03fe0d20990aad09c959be4be2f2a08caa74a423eff7d7cd4820"
Sep 11 17:54:20.410493 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:20.410453    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cba98995bebc03fe0d20990aad09c959be4be2f2a08caa74a423eff7d7cd4820"} err="failed to get container status \"cba98995bebc03fe0d20990aad09c959be4be2f2a08caa74a423eff7d7cd4820\": rpc error: code = NotFound desc = could not find container \"cba98995bebc03fe0d20990aad09c959be4be2f2a08caa74a423eff7d7cd4820\": container with ID starting with cba98995bebc03fe0d20990aad09c959be4be2f2a08caa74a423eff7d7cd4820 not found: ID does not exist"
Sep 11 17:54:20.417482 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:20.417461    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fac54-7dfb59d654-ntv7d"]
Sep 11 17:54:20.421289 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:20.421269    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fac54-7dfb59d654-ntv7d"]
Sep 11 17:54:22.126389 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:22.126353    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84c23adc-e901-4cc9-80d1-18e632314638" path="/var/lib/kubelet/pods/84c23adc-e901-4cc9-80d1-18e632314638/volumes"
Sep 11 17:54:26.384025 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:26.383972    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"]
Sep 11 17:54:26.384441 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:26.384202    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk" podUID="2faa8d27-577e-4dba-8d81-e5cd42ba7e02" containerName="istio-proxy" containerID="cri-o://0e8e3d6f522aaaf49aa9221950ab4684a3fd7b3eb81e15bbe8b92ee8f461254d" gracePeriod=30
Sep 11 17:54:31.632407 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:31.632378    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"
Sep 11 17:54:31.755654 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:31.755556    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-istio-token\") pod \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\" (UID: \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\") "
Sep 11 17:54:31.755654 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:31.755611    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-istio-data\") pod \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\" (UID: \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\") "
Sep 11 17:54:31.755654 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:31.755641    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-workload-socket\") pod \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\" (UID: \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\") "
Sep 11 17:54:31.755931 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:31.755670    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-credential-socket\") pod \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\" (UID: \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\") "
Sep 11 17:54:31.755931 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:31.755701    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-workload-certs\") pod \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\" (UID: \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\") "
Sep 11 17:54:31.755931 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:31.755741    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-istio-podinfo\") pod \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\" (UID: \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\") "
Sep 11 17:54:31.755931 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:31.755768    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-istio-envoy\") pod \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\" (UID: \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\") "
Sep 11 17:54:31.755931 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:31.755793    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlrnb\" (UniqueName: \"kubernetes.io/projected/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-kube-api-access-zlrnb\") pod \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\" (UID: \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\") "
Sep 11 17:54:31.755931 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:31.755880    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-istiod-ca-cert\") pod \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\" (UID: \"2faa8d27-577e-4dba-8d81-e5cd42ba7e02\") "
Sep 11 17:54:31.756311 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:31.755988    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-istio-data" (OuterVolumeSpecName: "istio-data") pod "2faa8d27-577e-4dba-8d81-e5cd42ba7e02" (UID: "2faa8d27-577e-4dba-8d81-e5cd42ba7e02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:54:31.756311 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:31.756033    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2faa8d27-577e-4dba-8d81-e5cd42ba7e02" (UID: "2faa8d27-577e-4dba-8d81-e5cd42ba7e02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:54:31.756311 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:31.756043    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2faa8d27-577e-4dba-8d81-e5cd42ba7e02" (UID: "2faa8d27-577e-4dba-8d81-e5cd42ba7e02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:54:31.756311 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:31.756124    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2faa8d27-577e-4dba-8d81-e5cd42ba7e02" (UID: "2faa8d27-577e-4dba-8d81-e5cd42ba7e02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:54:31.756532 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:31.756336    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-istio-data\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:54:31.756532 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:31.756363    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-workload-socket\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:54:31.756532 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:31.756370    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2faa8d27-577e-4dba-8d81-e5cd42ba7e02" (UID: "2faa8d27-577e-4dba-8d81-e5cd42ba7e02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:54:31.756532 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:31.756381    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-credential-socket\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:54:31.756532 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:31.756415    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-workload-certs\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:54:31.757963 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:31.757917    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-istio-token" (OuterVolumeSpecName: "istio-token") pod "2faa8d27-577e-4dba-8d81-e5cd42ba7e02" (UID: "2faa8d27-577e-4dba-8d81-e5cd42ba7e02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:54:31.757963 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:31.757953    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-kube-api-access-zlrnb" (OuterVolumeSpecName: "kube-api-access-zlrnb") pod "2faa8d27-577e-4dba-8d81-e5cd42ba7e02" (UID: "2faa8d27-577e-4dba-8d81-e5cd42ba7e02"). InnerVolumeSpecName "kube-api-access-zlrnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:54:31.758165 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:31.758043    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2faa8d27-577e-4dba-8d81-e5cd42ba7e02" (UID: "2faa8d27-577e-4dba-8d81-e5cd42ba7e02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 17:54:31.758165 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:31.758110    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2faa8d27-577e-4dba-8d81-e5cd42ba7e02" (UID: "2faa8d27-577e-4dba-8d81-e5cd42ba7e02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:54:31.857893 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:31.857849    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-istio-token\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:54:31.857893 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:31.857885    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-istio-podinfo\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:54:31.857893 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:31.857900    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-istio-envoy\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:54:31.858176 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:31.857912    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlrnb\" (UniqueName: \"kubernetes.io/projected/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-kube-api-access-zlrnb\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:54:31.858176 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:31.857925    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2faa8d27-577e-4dba-8d81-e5cd42ba7e02-istiod-ca-cert\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:54:32.458689 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:32.458592    2562 generic.go:356] "Generic (PLEG): container finished" podID="2faa8d27-577e-4dba-8d81-e5cd42ba7e02" containerID="0e8e3d6f522aaaf49aa9221950ab4684a3fd7b3eb81e15bbe8b92ee8f461254d" exitCode=0
Sep 11 17:54:32.458689 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:32.458639    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"
Sep 11 17:54:32.458942 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:32.458678    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk" event={"ID":"2faa8d27-577e-4dba-8d81-e5cd42ba7e02","Type":"ContainerDied","Data":"0e8e3d6f522aaaf49aa9221950ab4684a3fd7b3eb81e15bbe8b92ee8f461254d"}
Sep 11 17:54:32.458942 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:32.458744    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk" event={"ID":"2faa8d27-577e-4dba-8d81-e5cd42ba7e02","Type":"ContainerDied","Data":"401443cfd840afeb3134e47dea52d15466bcb8dbf20bd50a8443a727a7dcbe0a"}
Sep 11 17:54:32.458942 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:32.458775    2562 scope.go:117] "RemoveContainer" containerID="0e8e3d6f522aaaf49aa9221950ab4684a3fd7b3eb81e15bbe8b92ee8f461254d"
Sep 11 17:54:32.470811 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:32.470794    2562 scope.go:117] "RemoveContainer" containerID="e28076078c10dc6c00a3c8706d941f72b23e8f75cf9f17f83d41b0175cc506c2"
Sep 11 17:54:32.478670 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:32.478642    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"]
Sep 11 17:54:32.479778 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:32.479759    2562 scope.go:117] "RemoveContainer" containerID="0e8e3d6f522aaaf49aa9221950ab4684a3fd7b3eb81e15bbe8b92ee8f461254d"
Sep 11 17:54:32.480298 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:54:32.480217    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e8e3d6f522aaaf49aa9221950ab4684a3fd7b3eb81e15bbe8b92ee8f461254d\": container with ID starting with 0e8e3d6f522aaaf49aa9221950ab4684a3fd7b3eb81e15bbe8b92ee8f461254d not found: ID does not exist" containerID="0e8e3d6f522aaaf49aa9221950ab4684a3fd7b3eb81e15bbe8b92ee8f461254d"
Sep 11 17:54:32.480298 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:32.480259    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e8e3d6f522aaaf49aa9221950ab4684a3fd7b3eb81e15bbe8b92ee8f461254d"} err="failed to get container status \"0e8e3d6f522aaaf49aa9221950ab4684a3fd7b3eb81e15bbe8b92ee8f461254d\": rpc error: code = NotFound desc = could not find container \"0e8e3d6f522aaaf49aa9221950ab4684a3fd7b3eb81e15bbe8b92ee8f461254d\": container with ID starting with 0e8e3d6f522aaaf49aa9221950ab4684a3fd7b3eb81e15bbe8b92ee8f461254d not found: ID does not exist"
Sep 11 17:54:32.480298 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:32.480283    2562 scope.go:117] "RemoveContainer" containerID="e28076078c10dc6c00a3c8706d941f72b23e8f75cf9f17f83d41b0175cc506c2"
Sep 11 17:54:32.480661 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:54:32.480625    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e28076078c10dc6c00a3c8706d941f72b23e8f75cf9f17f83d41b0175cc506c2\": container with ID starting with e28076078c10dc6c00a3c8706d941f72b23e8f75cf9f17f83d41b0175cc506c2 not found: ID does not exist" containerID="e28076078c10dc6c00a3c8706d941f72b23e8f75cf9f17f83d41b0175cc506c2"
Sep 11 17:54:32.480743 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:32.480662    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e28076078c10dc6c00a3c8706d941f72b23e8f75cf9f17f83d41b0175cc506c2"} err="failed to get container status \"e28076078c10dc6c00a3c8706d941f72b23e8f75cf9f17f83d41b0175cc506c2\": rpc error: code = NotFound desc = could not find container \"e28076078c10dc6c00a3c8706d941f72b23e8f75cf9f17f83d41b0175cc506c2\": container with ID starting with e28076078c10dc6c00a3c8706d941f72b23e8f75cf9f17f83d41b0175cc506c2 not found: ID does not exist"
Sep 11 17:54:32.482359 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:32.482338    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2fac54-openshift-default-7bbdb755d6-b9hnk"]
Sep 11 17:54:34.125668 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:34.125633    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2faa8d27-577e-4dba-8d81-e5cd42ba7e02" path="/var/lib/kubelet/pods/2faa8d27-577e-4dba-8d81-e5cd42ba7e02/volumes"
Sep 11 17:54:37.390048 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:37.390012    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6fc9e6-c8c75bccf-b7vlq"]
Sep 11 17:54:37.390466 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:37.390282    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6fc9e6-c8c75bccf-b7vlq" podUID="b5db0334-d05c-4df6-9ccf-b833171869af" containerName="payload-processing" containerID="cri-o://8cad24b0987a2db41718cdfc70b9ee6ef91a2699a312a5ba59579960db18abd5" gracePeriod=30
Sep 11 17:54:37.631064 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:37.631040    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6fc9e6-c8c75bccf-b7vlq"
Sep 11 17:54:37.710535 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:37.710414    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5db0334-d05c-4df6-9ccf-b833171869af-plugins-config-volume\") pod \"b5db0334-d05c-4df6-9ccf-b833171869af\" (UID: \"b5db0334-d05c-4df6-9ccf-b833171869af\") "
Sep 11 17:54:37.710535 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:37.710479    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66qsx\" (UniqueName: \"kubernetes.io/projected/b5db0334-d05c-4df6-9ccf-b833171869af-kube-api-access-66qsx\") pod \"b5db0334-d05c-4df6-9ccf-b833171869af\" (UID: \"b5db0334-d05c-4df6-9ccf-b833171869af\") "
Sep 11 17:54:37.712344 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:37.712297    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5db0334-d05c-4df6-9ccf-b833171869af-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b5db0334-d05c-4df6-9ccf-b833171869af" (UID: "b5db0334-d05c-4df6-9ccf-b833171869af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:54:37.713636 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:37.713599    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5db0334-d05c-4df6-9ccf-b833171869af-kube-api-access-66qsx" (OuterVolumeSpecName: "kube-api-access-66qsx") pod "b5db0334-d05c-4df6-9ccf-b833171869af" (UID: "b5db0334-d05c-4df6-9ccf-b833171869af"). InnerVolumeSpecName "kube-api-access-66qsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:54:37.812176 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:37.812132    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5db0334-d05c-4df6-9ccf-b833171869af-plugins-config-volume\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:54:37.812176 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:37.812167    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66qsx\" (UniqueName: \"kubernetes.io/projected/b5db0334-d05c-4df6-9ccf-b833171869af-kube-api-access-66qsx\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:54:38.489139 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:38.489043    2562 generic.go:356] "Generic (PLEG): container finished" podID="b5db0334-d05c-4df6-9ccf-b833171869af" containerID="8cad24b0987a2db41718cdfc70b9ee6ef91a2699a312a5ba59579960db18abd5" exitCode=0
Sep 11 17:54:38.489139 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:38.489092    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6fc9e6-c8c75bccf-b7vlq"
Sep 11 17:54:38.489139 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:38.489104    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6fc9e6-c8c75bccf-b7vlq" event={"ID":"b5db0334-d05c-4df6-9ccf-b833171869af","Type":"ContainerDied","Data":"8cad24b0987a2db41718cdfc70b9ee6ef91a2699a312a5ba59579960db18abd5"}
Sep 11 17:54:38.489641 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:38.489157    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6fc9e6-c8c75bccf-b7vlq" event={"ID":"b5db0334-d05c-4df6-9ccf-b833171869af","Type":"ContainerDied","Data":"a076ad0437a2a2b4b50f9c719c425065c9bda2fed8029f4aeddf277ebc265d5b"}
Sep 11 17:54:38.489641 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:38.489188    2562 scope.go:117] "RemoveContainer" containerID="8cad24b0987a2db41718cdfc70b9ee6ef91a2699a312a5ba59579960db18abd5"
Sep 11 17:54:38.497581 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:38.497562    2562 scope.go:117] "RemoveContainer" containerID="8cad24b0987a2db41718cdfc70b9ee6ef91a2699a312a5ba59579960db18abd5"
Sep 11 17:54:38.497868 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:54:38.497847    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cad24b0987a2db41718cdfc70b9ee6ef91a2699a312a5ba59579960db18abd5\": container with ID starting with 8cad24b0987a2db41718cdfc70b9ee6ef91a2699a312a5ba59579960db18abd5 not found: ID does not exist" containerID="8cad24b0987a2db41718cdfc70b9ee6ef91a2699a312a5ba59579960db18abd5"
Sep 11 17:54:38.497981 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:38.497877    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cad24b0987a2db41718cdfc70b9ee6ef91a2699a312a5ba59579960db18abd5"} err="failed to get container status \"8cad24b0987a2db41718cdfc70b9ee6ef91a2699a312a5ba59579960db18abd5\": rpc error: code = NotFound desc = could not find container \"8cad24b0987a2db41718cdfc70b9ee6ef91a2699a312a5ba59579960db18abd5\": container with ID starting with 8cad24b0987a2db41718cdfc70b9ee6ef91a2699a312a5ba59579960db18abd5 not found: ID does not exist"
Sep 11 17:54:38.514334 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:38.514310    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6fc9e6-c8c75bccf-b7vlq"]
Sep 11 17:54:38.517032 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:38.517009    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6fc9e6-c8c75bccf-b7vlq"]
Sep 11 17:54:40.125569 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:40.125536    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5db0334-d05c-4df6-9ccf-b833171869af" path="/var/lib/kubelet/pods/b5db0334-d05c-4df6-9ccf-b833171869af/volumes"
Sep 11 17:54:40.392865 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:40.392783    2562 scope.go:117] "RemoveContainer" containerID="18995f2c1ae77c9a47884adb47dbc5d3d44f15b454ed2ac8b501bbebc46c1d88"
Sep 11 17:54:46.860508 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:46.860469    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"]
Sep 11 17:54:46.860939 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:46.860697    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5" podUID="7868f970-d989-409a-bf67-f184f8e39ca0" containerName="istio-proxy" containerID="cri-o://2d70a58ff0d680f67455c7fd24e0f4507061bfd3f17043e28db2f6fd88eae4d4" gracePeriod=30
Sep 11 17:54:49.985534 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:49.985494    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7549975874-5snr4"]
Sep 11 17:54:49.986092 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:49.985702    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7549975874-5snr4" podUID="f08ec6ab-9dd1-412d-a85c-4a3802f2f895" containerName="manager" containerID="cri-o://e489ce91f67545b56b8d66f1eaa22b388f8fc35f10b1f5bd47f92ca3f0412fac" gracePeriod=30
Sep 11 17:54:50.231132 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:50.231103    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7549975874-5snr4"
Sep 11 17:54:50.326446 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:50.326362    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f08ec6ab-9dd1-412d-a85c-4a3802f2f895-webhook-cert\") pod \"f08ec6ab-9dd1-412d-a85c-4a3802f2f895\" (UID: \"f08ec6ab-9dd1-412d-a85c-4a3802f2f895\") "
Sep 11 17:54:50.326446 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:50.326406    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f08ec6ab-9dd1-412d-a85c-4a3802f2f895-metrics-certs\") pod \"f08ec6ab-9dd1-412d-a85c-4a3802f2f895\" (UID: \"f08ec6ab-9dd1-412d-a85c-4a3802f2f895\") "
Sep 11 17:54:50.326677 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:50.326452    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmzl4\" (UniqueName: \"kubernetes.io/projected/f08ec6ab-9dd1-412d-a85c-4a3802f2f895-kube-api-access-pmzl4\") pod \"f08ec6ab-9dd1-412d-a85c-4a3802f2f895\" (UID: \"f08ec6ab-9dd1-412d-a85c-4a3802f2f895\") "
Sep 11 17:54:50.328499 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:50.328465    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f08ec6ab-9dd1-412d-a85c-4a3802f2f895-kube-api-access-pmzl4" (OuterVolumeSpecName: "kube-api-access-pmzl4") pod "f08ec6ab-9dd1-412d-a85c-4a3802f2f895" (UID: "f08ec6ab-9dd1-412d-a85c-4a3802f2f895"). InnerVolumeSpecName "kube-api-access-pmzl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:54:50.328499 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:50.328484    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f08ec6ab-9dd1-412d-a85c-4a3802f2f895-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f08ec6ab-9dd1-412d-a85c-4a3802f2f895" (UID: "f08ec6ab-9dd1-412d-a85c-4a3802f2f895"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:54:50.328677 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:50.328504    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f08ec6ab-9dd1-412d-a85c-4a3802f2f895-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f08ec6ab-9dd1-412d-a85c-4a3802f2f895" (UID: "f08ec6ab-9dd1-412d-a85c-4a3802f2f895"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:54:50.427943 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:50.427913    2562 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f08ec6ab-9dd1-412d-a85c-4a3802f2f895-webhook-cert\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:54:50.427943 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:50.427937    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f08ec6ab-9dd1-412d-a85c-4a3802f2f895-metrics-certs\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:54:50.427943 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:50.427947    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmzl4\" (UniqueName: \"kubernetes.io/projected/f08ec6ab-9dd1-412d-a85c-4a3802f2f895-kube-api-access-pmzl4\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:54:50.545283 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:50.545250    2562 generic.go:356] "Generic (PLEG): container finished" podID="f08ec6ab-9dd1-412d-a85c-4a3802f2f895" containerID="e489ce91f67545b56b8d66f1eaa22b388f8fc35f10b1f5bd47f92ca3f0412fac" exitCode=0
Sep 11 17:54:50.545448 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:50.545296    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7549975874-5snr4"
Sep 11 17:54:50.545448 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:50.545327    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7549975874-5snr4" event={"ID":"f08ec6ab-9dd1-412d-a85c-4a3802f2f895","Type":"ContainerDied","Data":"e489ce91f67545b56b8d66f1eaa22b388f8fc35f10b1f5bd47f92ca3f0412fac"}
Sep 11 17:54:50.545448 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:50.545366    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7549975874-5snr4" event={"ID":"f08ec6ab-9dd1-412d-a85c-4a3802f2f895","Type":"ContainerDied","Data":"e3eb3a02658ecf80187b8e2cfa2754c4fde5efed8089a5478872e5074afb8451"}
Sep 11 17:54:50.545448 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:50.545388    2562 scope.go:117] "RemoveContainer" containerID="e489ce91f67545b56b8d66f1eaa22b388f8fc35f10b1f5bd47f92ca3f0412fac"
Sep 11 17:54:50.554806 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:50.554788    2562 scope.go:117] "RemoveContainer" containerID="e489ce91f67545b56b8d66f1eaa22b388f8fc35f10b1f5bd47f92ca3f0412fac"
Sep 11 17:54:50.555080 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:54:50.555060    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e489ce91f67545b56b8d66f1eaa22b388f8fc35f10b1f5bd47f92ca3f0412fac\": container with ID starting with e489ce91f67545b56b8d66f1eaa22b388f8fc35f10b1f5bd47f92ca3f0412fac not found: ID does not exist" containerID="e489ce91f67545b56b8d66f1eaa22b388f8fc35f10b1f5bd47f92ca3f0412fac"
Sep 11 17:54:50.555125 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:50.555088    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e489ce91f67545b56b8d66f1eaa22b388f8fc35f10b1f5bd47f92ca3f0412fac"} err="failed to get container status \"e489ce91f67545b56b8d66f1eaa22b388f8fc35f10b1f5bd47f92ca3f0412fac\": rpc error: code = NotFound desc = could not find container \"e489ce91f67545b56b8d66f1eaa22b388f8fc35f10b1f5bd47f92ca3f0412fac\": container with ID starting with e489ce91f67545b56b8d66f1eaa22b388f8fc35f10b1f5bd47f92ca3f0412fac not found: ID does not exist"
Sep 11 17:54:50.584466 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:50.584397    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7549975874-5snr4"]
Sep 11 17:54:50.590014 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:50.589977    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7549975874-5snr4"]
Sep 11 17:54:52.107198 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.107166    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"
Sep 11 17:54:52.126838 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.126746    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f08ec6ab-9dd1-412d-a85c-4a3802f2f895" path="/var/lib/kubelet/pods/f08ec6ab-9dd1-412d-a85c-4a3802f2f895/volumes"
Sep 11 17:54:52.243919 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.243871    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7868f970-d989-409a-bf67-f184f8e39ca0-workload-certs\") pod \"7868f970-d989-409a-bf67-f184f8e39ca0\" (UID: \"7868f970-d989-409a-bf67-f184f8e39ca0\") "
Sep 11 17:54:52.244124 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.243946    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7868f970-d989-409a-bf67-f184f8e39ca0-credential-socket\") pod \"7868f970-d989-409a-bf67-f184f8e39ca0\" (UID: \"7868f970-d989-409a-bf67-f184f8e39ca0\") "
Sep 11 17:54:52.244124 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.243984    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7868f970-d989-409a-bf67-f184f8e39ca0-istio-token\") pod \"7868f970-d989-409a-bf67-f184f8e39ca0\" (UID: \"7868f970-d989-409a-bf67-f184f8e39ca0\") "
Sep 11 17:54:52.244124 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.244051    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdlrs\" (UniqueName: \"kubernetes.io/projected/7868f970-d989-409a-bf67-f184f8e39ca0-kube-api-access-wdlrs\") pod \"7868f970-d989-409a-bf67-f184f8e39ca0\" (UID: \"7868f970-d989-409a-bf67-f184f8e39ca0\") "
Sep 11 17:54:52.244124 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.244087    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7868f970-d989-409a-bf67-f184f8e39ca0-istio-data\") pod \"7868f970-d989-409a-bf67-f184f8e39ca0\" (UID: \"7868f970-d989-409a-bf67-f184f8e39ca0\") "
Sep 11 17:54:52.244282 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.244190    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7868f970-d989-409a-bf67-f184f8e39ca0-workload-socket\") pod \"7868f970-d989-409a-bf67-f184f8e39ca0\" (UID: \"7868f970-d989-409a-bf67-f184f8e39ca0\") "
Sep 11 17:54:52.244282 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.244223    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7868f970-d989-409a-bf67-f184f8e39ca0-istio-podinfo\") pod \"7868f970-d989-409a-bf67-f184f8e39ca0\" (UID: \"7868f970-d989-409a-bf67-f184f8e39ca0\") "
Sep 11 17:54:52.244282 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.244248    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7868f970-d989-409a-bf67-f184f8e39ca0-istiod-ca-cert\") pod \"7868f970-d989-409a-bf67-f184f8e39ca0\" (UID: \"7868f970-d989-409a-bf67-f184f8e39ca0\") "
Sep 11 17:54:52.244378 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.244284    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7868f970-d989-409a-bf67-f184f8e39ca0-istio-envoy\") pod \"7868f970-d989-409a-bf67-f184f8e39ca0\" (UID: \"7868f970-d989-409a-bf67-f184f8e39ca0\") "
Sep 11 17:54:52.244378 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.244293    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7868f970-d989-409a-bf67-f184f8e39ca0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7868f970-d989-409a-bf67-f184f8e39ca0" (UID: "7868f970-d989-409a-bf67-f184f8e39ca0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:54:52.244378 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.244323    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7868f970-d989-409a-bf67-f184f8e39ca0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7868f970-d989-409a-bf67-f184f8e39ca0" (UID: "7868f970-d989-409a-bf67-f184f8e39ca0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:54:52.244556 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.244534    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7868f970-d989-409a-bf67-f184f8e39ca0-istio-data" (OuterVolumeSpecName: "istio-data") pod "7868f970-d989-409a-bf67-f184f8e39ca0" (UID: "7868f970-d989-409a-bf67-f184f8e39ca0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:54:52.244593 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.244548    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7868f970-d989-409a-bf67-f184f8e39ca0-workload-certs\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:54:52.244593 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.244584    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7868f970-d989-409a-bf67-f184f8e39ca0-credential-socket\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:54:52.244983 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.244940    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7868f970-d989-409a-bf67-f184f8e39ca0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7868f970-d989-409a-bf67-f184f8e39ca0" (UID: "7868f970-d989-409a-bf67-f184f8e39ca0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:54:52.245163 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.245078    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7868f970-d989-409a-bf67-f184f8e39ca0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7868f970-d989-409a-bf67-f184f8e39ca0" (UID: "7868f970-d989-409a-bf67-f184f8e39ca0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:54:52.246815 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.246780    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7868f970-d989-409a-bf67-f184f8e39ca0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7868f970-d989-409a-bf67-f184f8e39ca0" (UID: "7868f970-d989-409a-bf67-f184f8e39ca0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 17:54:52.247159 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.247126    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7868f970-d989-409a-bf67-f184f8e39ca0-istio-token" (OuterVolumeSpecName: "istio-token") pod "7868f970-d989-409a-bf67-f184f8e39ca0" (UID: "7868f970-d989-409a-bf67-f184f8e39ca0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:54:52.247390 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.247355    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7868f970-d989-409a-bf67-f184f8e39ca0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7868f970-d989-409a-bf67-f184f8e39ca0" (UID: "7868f970-d989-409a-bf67-f184f8e39ca0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:54:52.247509 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.247399    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7868f970-d989-409a-bf67-f184f8e39ca0-kube-api-access-wdlrs" (OuterVolumeSpecName: "kube-api-access-wdlrs") pod "7868f970-d989-409a-bf67-f184f8e39ca0" (UID: "7868f970-d989-409a-bf67-f184f8e39ca0"). InnerVolumeSpecName "kube-api-access-wdlrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:54:52.345198 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.345166    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7868f970-d989-409a-bf67-f184f8e39ca0-istio-data\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:54:52.345198 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.345191    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7868f970-d989-409a-bf67-f184f8e39ca0-workload-socket\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:54:52.345198 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.345200    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7868f970-d989-409a-bf67-f184f8e39ca0-istio-podinfo\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:54:52.345198 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.345208    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7868f970-d989-409a-bf67-f184f8e39ca0-istiod-ca-cert\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:54:52.345590 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.345218    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7868f970-d989-409a-bf67-f184f8e39ca0-istio-envoy\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:54:52.345590 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.345225    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7868f970-d989-409a-bf67-f184f8e39ca0-istio-token\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:54:52.345590 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.345233    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdlrs\" (UniqueName: \"kubernetes.io/projected/7868f970-d989-409a-bf67-f184f8e39ca0-kube-api-access-wdlrs\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:54:52.556180 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.556086    2562 generic.go:356] "Generic (PLEG): container finished" podID="7868f970-d989-409a-bf67-f184f8e39ca0" containerID="2d70a58ff0d680f67455c7fd24e0f4507061bfd3f17043e28db2f6fd88eae4d4" exitCode=0
Sep 11 17:54:52.556180 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.556131    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5" event={"ID":"7868f970-d989-409a-bf67-f184f8e39ca0","Type":"ContainerDied","Data":"2d70a58ff0d680f67455c7fd24e0f4507061bfd3f17043e28db2f6fd88eae4d4"}
Sep 11 17:54:52.556180 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.556136    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"
Sep 11 17:54:52.556481 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.556183    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5" event={"ID":"7868f970-d989-409a-bf67-f184f8e39ca0","Type":"ContainerDied","Data":"d5aa06ebf46d42f65fe315e078d1a2c9b7f8eaa50647d039ad222978fc9921ca"}
Sep 11 17:54:52.556481 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.556214    2562 scope.go:117] "RemoveContainer" containerID="2d70a58ff0d680f67455c7fd24e0f4507061bfd3f17043e28db2f6fd88eae4d4"
Sep 11 17:54:52.565553 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.565533    2562 scope.go:117] "RemoveContainer" containerID="c361f1de4c44da0554bcfdcc08dd2351e1ed4efe418d5be2c10d1d25e6409a09"
Sep 11 17:54:52.573165 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.573149    2562 scope.go:117] "RemoveContainer" containerID="2d70a58ff0d680f67455c7fd24e0f4507061bfd3f17043e28db2f6fd88eae4d4"
Sep 11 17:54:52.573429 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:54:52.573411    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d70a58ff0d680f67455c7fd24e0f4507061bfd3f17043e28db2f6fd88eae4d4\": container with ID starting with 2d70a58ff0d680f67455c7fd24e0f4507061bfd3f17043e28db2f6fd88eae4d4 not found: ID does not exist" containerID="2d70a58ff0d680f67455c7fd24e0f4507061bfd3f17043e28db2f6fd88eae4d4"
Sep 11 17:54:52.573480 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.573437    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d70a58ff0d680f67455c7fd24e0f4507061bfd3f17043e28db2f6fd88eae4d4"} err="failed to get container status \"2d70a58ff0d680f67455c7fd24e0f4507061bfd3f17043e28db2f6fd88eae4d4\": rpc error: code = NotFound desc = could not find container \"2d70a58ff0d680f67455c7fd24e0f4507061bfd3f17043e28db2f6fd88eae4d4\": container with ID starting with 2d70a58ff0d680f67455c7fd24e0f4507061bfd3f17043e28db2f6fd88eae4d4 not found: ID does not exist"
Sep 11 17:54:52.573480 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.573455    2562 scope.go:117] "RemoveContainer" containerID="c361f1de4c44da0554bcfdcc08dd2351e1ed4efe418d5be2c10d1d25e6409a09"
Sep 11 17:54:52.573701 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:54:52.573682    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c361f1de4c44da0554bcfdcc08dd2351e1ed4efe418d5be2c10d1d25e6409a09\": container with ID starting with c361f1de4c44da0554bcfdcc08dd2351e1ed4efe418d5be2c10d1d25e6409a09 not found: ID does not exist" containerID="c361f1de4c44da0554bcfdcc08dd2351e1ed4efe418d5be2c10d1d25e6409a09"
Sep 11 17:54:52.573736 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.573708    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c361f1de4c44da0554bcfdcc08dd2351e1ed4efe418d5be2c10d1d25e6409a09"} err="failed to get container status \"c361f1de4c44da0554bcfdcc08dd2351e1ed4efe418d5be2c10d1d25e6409a09\": rpc error: code = NotFound desc = could not find container \"c361f1de4c44da0554bcfdcc08dd2351e1ed4efe418d5be2c10d1d25e6409a09\": container with ID starting with c361f1de4c44da0554bcfdcc08dd2351e1ed4efe418d5be2c10d1d25e6409a09 not found: ID does not exist"
Sep 11 17:54:52.585400 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.585378    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"]
Sep 11 17:54:52.586972 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:52.586952    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6fc9e6-openshift-default-7bdcf8c784-hvdc5"]
Sep 11 17:54:54.129953 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:54:54.126747    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7868f970-d989-409a-bf67-f184f8e39ca0" path="/var/lib/kubelet/pods/7868f970-d989-409a-bf67-f184f8e39ca0/volumes"
Sep 11 17:55:51.410010 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.409963    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7549975874-fdrlg"]
Sep 11 17:55:51.410469 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.410302    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2da9345-6535-464d-9048-0e6008496838" containerName="istio-proxy"
Sep 11 17:55:51.410469 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.410314    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2da9345-6535-464d-9048-0e6008496838" containerName="istio-proxy"
Sep 11 17:55:51.410469 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.410324    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7868f970-d989-409a-bf67-f184f8e39ca0" containerName="istio-proxy"
Sep 11 17:55:51.410469 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.410330    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7868f970-d989-409a-bf67-f184f8e39ca0" containerName="istio-proxy"
Sep 11 17:55:51.410469 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.410341    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a52eaa4-863e-4526-8ccd-aff1a26fb1b4" containerName="payload-pre-processing"
Sep 11 17:55:51.410469 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.410348    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a52eaa4-863e-4526-8ccd-aff1a26fb1b4" containerName="payload-pre-processing"
Sep 11 17:55:51.410469 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.410360    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="163e963d-b4a7-40c2-b6a9-5df5b848e182" containerName="payload-processing"
Sep 11 17:55:51.410469 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.410367    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="163e963d-b4a7-40c2-b6a9-5df5b848e182" containerName="payload-processing"
Sep 11 17:55:51.410469 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.410374    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f08ec6ab-9dd1-412d-a85c-4a3802f2f895" containerName="manager"
Sep 11 17:55:51.410469 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.410379    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f08ec6ab-9dd1-412d-a85c-4a3802f2f895" containerName="manager"
Sep 11 17:55:51.410469 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.410386    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84c23adc-e901-4cc9-80d1-18e632314638" containerName="payload-pre-processing"
Sep 11 17:55:51.410469 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.410391    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="84c23adc-e901-4cc9-80d1-18e632314638" containerName="payload-pre-processing"
Sep 11 17:55:51.410469 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.410398    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5db0334-d05c-4df6-9ccf-b833171869af" containerName="payload-processing"
Sep 11 17:55:51.410469 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.410403    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5db0334-d05c-4df6-9ccf-b833171869af" containerName="payload-processing"
Sep 11 17:55:51.410469 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.410411    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7868f970-d989-409a-bf67-f184f8e39ca0" containerName="istio-proxy"
Sep 11 17:55:51.410469 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.410421    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7868f970-d989-409a-bf67-f184f8e39ca0" containerName="istio-proxy"
Sep 11 17:55:51.410469 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.410430    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2faa8d27-577e-4dba-8d81-e5cd42ba7e02" containerName="istio-proxy"
Sep 11 17:55:51.410469 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.410435    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2faa8d27-577e-4dba-8d81-e5cd42ba7e02" containerName="istio-proxy"
Sep 11 17:55:51.410469 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.410441    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2faa8d27-577e-4dba-8d81-e5cd42ba7e02" containerName="istio-proxy"
Sep 11 17:55:51.410469 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.410446    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2faa8d27-577e-4dba-8d81-e5cd42ba7e02" containerName="istio-proxy"
Sep 11 17:55:51.411187 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.410517    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f08ec6ab-9dd1-412d-a85c-4a3802f2f895" containerName="manager"
Sep 11 17:55:51.411187 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.410524    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2da9345-6535-464d-9048-0e6008496838" containerName="istio-proxy"
Sep 11 17:55:51.411187 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.410529    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2da9345-6535-464d-9048-0e6008496838" containerName="istio-proxy"
Sep 11 17:55:51.411187 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.410536    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a52eaa4-863e-4526-8ccd-aff1a26fb1b4" containerName="payload-pre-processing"
Sep 11 17:55:51.411187 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.410542    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7868f970-d989-409a-bf67-f184f8e39ca0" containerName="istio-proxy"
Sep 11 17:55:51.411187 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.410548    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5db0334-d05c-4df6-9ccf-b833171869af" containerName="payload-processing"
Sep 11 17:55:51.411187 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.410553    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7868f970-d989-409a-bf67-f184f8e39ca0" containerName="istio-proxy"
Sep 11 17:55:51.411187 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.410559    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="84c23adc-e901-4cc9-80d1-18e632314638" containerName="payload-pre-processing"
Sep 11 17:55:51.411187 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.410566    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="163e963d-b4a7-40c2-b6a9-5df5b848e182" containerName="payload-processing"
Sep 11 17:55:51.411187 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.410575    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2faa8d27-577e-4dba-8d81-e5cd42ba7e02" containerName="istio-proxy"
Sep 11 17:55:51.411187 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.410581    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2faa8d27-577e-4dba-8d81-e5cd42ba7e02" containerName="istio-proxy"
Sep 11 17:55:51.413537 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.413520    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7549975874-fdrlg"
Sep 11 17:55:51.416391 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.416372    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 17:55:51.416742 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.416719    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-thh5n\""
Sep 11 17:55:51.417187 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.417169    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 17:55:51.418190 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.418175    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 17:55:51.425296 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.425268    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7549975874-fdrlg"]
Sep 11 17:55:51.543395 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.543354    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4vsc\" (UniqueName: \"kubernetes.io/projected/939a6abe-d5a0-4a6b-ba68-a71ab3758c7a-kube-api-access-f4vsc\") pod \"maas-controller-7549975874-fdrlg\" (UID: \"939a6abe-d5a0-4a6b-ba68-a71ab3758c7a\") " pod="opendatahub/maas-controller-7549975874-fdrlg"
Sep 11 17:55:51.543593 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.543401    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/939a6abe-d5a0-4a6b-ba68-a71ab3758c7a-webhook-cert\") pod \"maas-controller-7549975874-fdrlg\" (UID: \"939a6abe-d5a0-4a6b-ba68-a71ab3758c7a\") " pod="opendatahub/maas-controller-7549975874-fdrlg"
Sep 11 17:55:51.543593 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.543503    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/939a6abe-d5a0-4a6b-ba68-a71ab3758c7a-metrics-certs\") pod \"maas-controller-7549975874-fdrlg\" (UID: \"939a6abe-d5a0-4a6b-ba68-a71ab3758c7a\") " pod="opendatahub/maas-controller-7549975874-fdrlg"
Sep 11 17:55:51.644819 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.644785    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4vsc\" (UniqueName: \"kubernetes.io/projected/939a6abe-d5a0-4a6b-ba68-a71ab3758c7a-kube-api-access-f4vsc\") pod \"maas-controller-7549975874-fdrlg\" (UID: \"939a6abe-d5a0-4a6b-ba68-a71ab3758c7a\") " pod="opendatahub/maas-controller-7549975874-fdrlg"
Sep 11 17:55:51.644973 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.644826    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/939a6abe-d5a0-4a6b-ba68-a71ab3758c7a-webhook-cert\") pod \"maas-controller-7549975874-fdrlg\" (UID: \"939a6abe-d5a0-4a6b-ba68-a71ab3758c7a\") " pod="opendatahub/maas-controller-7549975874-fdrlg"
Sep 11 17:55:51.644973 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.644902    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/939a6abe-d5a0-4a6b-ba68-a71ab3758c7a-metrics-certs\") pod \"maas-controller-7549975874-fdrlg\" (UID: \"939a6abe-d5a0-4a6b-ba68-a71ab3758c7a\") " pod="opendatahub/maas-controller-7549975874-fdrlg"
Sep 11 17:55:51.647402 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.647370    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/939a6abe-d5a0-4a6b-ba68-a71ab3758c7a-webhook-cert\") pod \"maas-controller-7549975874-fdrlg\" (UID: \"939a6abe-d5a0-4a6b-ba68-a71ab3758c7a\") " pod="opendatahub/maas-controller-7549975874-fdrlg"
Sep 11 17:55:51.647514 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.647439    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/939a6abe-d5a0-4a6b-ba68-a71ab3758c7a-metrics-certs\") pod \"maas-controller-7549975874-fdrlg\" (UID: \"939a6abe-d5a0-4a6b-ba68-a71ab3758c7a\") " pod="opendatahub/maas-controller-7549975874-fdrlg"
Sep 11 17:55:51.655567 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.655543    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4vsc\" (UniqueName: \"kubernetes.io/projected/939a6abe-d5a0-4a6b-ba68-a71ab3758c7a-kube-api-access-f4vsc\") pod \"maas-controller-7549975874-fdrlg\" (UID: \"939a6abe-d5a0-4a6b-ba68-a71ab3758c7a\") " pod="opendatahub/maas-controller-7549975874-fdrlg"
Sep 11 17:55:51.724644 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.724556    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7549975874-fdrlg"
Sep 11 17:55:51.997314 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:51.997288    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7549975874-fdrlg"]
Sep 11 17:55:51.999724 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:55:51.999700    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod939a6abe_d5a0_4a6b_ba68_a71ab3758c7a.slice/crio-62682ad590e306c2db8d4776adea92863cee1e012ace63f6e534876a9280575f WatchSource:0}: Error finding container 62682ad590e306c2db8d4776adea92863cee1e012ace63f6e534876a9280575f: Status 404 returned error can't find the container with id 62682ad590e306c2db8d4776adea92863cee1e012ace63f6e534876a9280575f
Sep 11 17:55:52.000962 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:52.000947    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 17:55:52.815544 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:52.815451    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7549975874-fdrlg" event={"ID":"939a6abe-d5a0-4a6b-ba68-a71ab3758c7a","Type":"ContainerStarted","Data":"f38df0d416c70ffa3f5ee6e87a4be7fc16e420affabcb8f9bd04a82975147bed"}
Sep 11 17:55:52.815906 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:52.815554    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7549975874-fdrlg"
Sep 11 17:55:52.815906 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:52.815573    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7549975874-fdrlg" event={"ID":"939a6abe-d5a0-4a6b-ba68-a71ab3758c7a","Type":"ContainerStarted","Data":"62682ad590e306c2db8d4776adea92863cee1e012ace63f6e534876a9280575f"}
Sep 11 17:55:52.853323 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:55:52.853273    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7549975874-fdrlg" podStartSLOduration=1.352052659 podStartE2EDuration="1.853256932s" totalImagesPullingTime="501.204273ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:55:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:52.852812229 +0000 UTC m=+2233.246729684" watchObservedRunningTime="2026-09-11 17:55:52.853256932 +0000 UTC m=+2233.247174387"
Sep 11 17:56:03.823544 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:03.823464    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7549975874-fdrlg"
Sep 11 17:56:13.181850 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:13.181816    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7549975874-fdrlg"]
Sep 11 17:56:13.182354 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:13.182053    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7549975874-fdrlg" podUID="939a6abe-d5a0-4a6b-ba68-a71ab3758c7a" containerName="manager" containerID="cri-o://f38df0d416c70ffa3f5ee6e87a4be7fc16e420affabcb8f9bd04a82975147bed" gracePeriod=30
Sep 11 17:56:13.431515 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:13.431490    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7549975874-fdrlg"
Sep 11 17:56:13.538935 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:13.538846    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4vsc\" (UniqueName: \"kubernetes.io/projected/939a6abe-d5a0-4a6b-ba68-a71ab3758c7a-kube-api-access-f4vsc\") pod \"939a6abe-d5a0-4a6b-ba68-a71ab3758c7a\" (UID: \"939a6abe-d5a0-4a6b-ba68-a71ab3758c7a\") "
Sep 11 17:56:13.538935 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:13.538883    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/939a6abe-d5a0-4a6b-ba68-a71ab3758c7a-metrics-certs\") pod \"939a6abe-d5a0-4a6b-ba68-a71ab3758c7a\" (UID: \"939a6abe-d5a0-4a6b-ba68-a71ab3758c7a\") "
Sep 11 17:56:13.538935 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:13.538921    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/939a6abe-d5a0-4a6b-ba68-a71ab3758c7a-webhook-cert\") pod \"939a6abe-d5a0-4a6b-ba68-a71ab3758c7a\" (UID: \"939a6abe-d5a0-4a6b-ba68-a71ab3758c7a\") "
Sep 11 17:56:13.540985 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:13.540951    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/939a6abe-d5a0-4a6b-ba68-a71ab3758c7a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "939a6abe-d5a0-4a6b-ba68-a71ab3758c7a" (UID: "939a6abe-d5a0-4a6b-ba68-a71ab3758c7a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:56:13.541135 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:13.541019    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/939a6abe-d5a0-4a6b-ba68-a71ab3758c7a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "939a6abe-d5a0-4a6b-ba68-a71ab3758c7a" (UID: "939a6abe-d5a0-4a6b-ba68-a71ab3758c7a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:56:13.541135 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:13.541024    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/939a6abe-d5a0-4a6b-ba68-a71ab3758c7a-kube-api-access-f4vsc" (OuterVolumeSpecName: "kube-api-access-f4vsc") pod "939a6abe-d5a0-4a6b-ba68-a71ab3758c7a" (UID: "939a6abe-d5a0-4a6b-ba68-a71ab3758c7a"). InnerVolumeSpecName "kube-api-access-f4vsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:56:13.640269 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:13.640229    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4vsc\" (UniqueName: \"kubernetes.io/projected/939a6abe-d5a0-4a6b-ba68-a71ab3758c7a-kube-api-access-f4vsc\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:56:13.640269 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:13.640266    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/939a6abe-d5a0-4a6b-ba68-a71ab3758c7a-metrics-certs\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:56:13.640497 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:13.640280    2562 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/939a6abe-d5a0-4a6b-ba68-a71ab3758c7a-webhook-cert\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 17:56:13.913456 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:13.913418    2562 generic.go:356] "Generic (PLEG): container finished" podID="939a6abe-d5a0-4a6b-ba68-a71ab3758c7a" containerID="f38df0d416c70ffa3f5ee6e87a4be7fc16e420affabcb8f9bd04a82975147bed" exitCode=0
Sep 11 17:56:13.913668 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:13.913471    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7549975874-fdrlg"
Sep 11 17:56:13.913668 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:13.913503    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7549975874-fdrlg" event={"ID":"939a6abe-d5a0-4a6b-ba68-a71ab3758c7a","Type":"ContainerDied","Data":"f38df0d416c70ffa3f5ee6e87a4be7fc16e420affabcb8f9bd04a82975147bed"}
Sep 11 17:56:13.913668 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:13.913566    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7549975874-fdrlg" event={"ID":"939a6abe-d5a0-4a6b-ba68-a71ab3758c7a","Type":"ContainerDied","Data":"62682ad590e306c2db8d4776adea92863cee1e012ace63f6e534876a9280575f"}
Sep 11 17:56:13.913668 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:13.913594    2562 scope.go:117] "RemoveContainer" containerID="f38df0d416c70ffa3f5ee6e87a4be7fc16e420affabcb8f9bd04a82975147bed"
Sep 11 17:56:13.922119 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:13.922101    2562 scope.go:117] "RemoveContainer" containerID="f38df0d416c70ffa3f5ee6e87a4be7fc16e420affabcb8f9bd04a82975147bed"
Sep 11 17:56:13.922352 ip-10-0-133-252 kubenswrapper[2562]: E0911 17:56:13.922334    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f38df0d416c70ffa3f5ee6e87a4be7fc16e420affabcb8f9bd04a82975147bed\": container with ID starting with f38df0d416c70ffa3f5ee6e87a4be7fc16e420affabcb8f9bd04a82975147bed not found: ID does not exist" containerID="f38df0d416c70ffa3f5ee6e87a4be7fc16e420affabcb8f9bd04a82975147bed"
Sep 11 17:56:13.922394 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:13.922366    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f38df0d416c70ffa3f5ee6e87a4be7fc16e420affabcb8f9bd04a82975147bed"} err="failed to get container status \"f38df0d416c70ffa3f5ee6e87a4be7fc16e420affabcb8f9bd04a82975147bed\": rpc error: code = NotFound desc = could not find container \"f38df0d416c70ffa3f5ee6e87a4be7fc16e420affabcb8f9bd04a82975147bed\": container with ID starting with f38df0d416c70ffa3f5ee6e87a4be7fc16e420affabcb8f9bd04a82975147bed not found: ID does not exist"
Sep 11 17:56:13.934119 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:13.934097    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7549975874-fdrlg"]
Sep 11 17:56:13.941138 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:13.941119    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7549975874-fdrlg"]
Sep 11 17:56:14.125451 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:14.125419    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="939a6abe-d5a0-4a6b-ba68-a71ab3758c7a" path="/var/lib/kubelet/pods/939a6abe-d5a0-4a6b-ba68-a71ab3758c7a/volumes"
Sep 11 17:56:14.724696 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:14.724662    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7549975874-7hq4b"]
Sep 11 17:56:14.725120 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:14.724962    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2da9345-6535-464d-9048-0e6008496838" containerName="istio-proxy"
Sep 11 17:56:14.725120 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:14.724973    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2da9345-6535-464d-9048-0e6008496838" containerName="istio-proxy"
Sep 11 17:56:14.725120 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:14.725005    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="939a6abe-d5a0-4a6b-ba68-a71ab3758c7a" containerName="manager"
Sep 11 17:56:14.725120 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:14.725014    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="939a6abe-d5a0-4a6b-ba68-a71ab3758c7a" containerName="manager"
Sep 11 17:56:14.725120 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:14.725085    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="939a6abe-d5a0-4a6b-ba68-a71ab3758c7a" containerName="manager"
Sep 11 17:56:14.729282 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:14.729260    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7549975874-7hq4b"
Sep 11 17:56:14.732394 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:14.732365    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 17:56:14.732530 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:14.732397    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 17:56:14.732530 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:14.732368    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-thh5n\""
Sep 11 17:56:14.732530 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:14.732525    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 17:56:14.738249 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:14.738226    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7549975874-7hq4b"]
Sep 11 17:56:14.851252 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:14.851219    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz9ld\" (UniqueName: \"kubernetes.io/projected/8a80a8ca-48ed-40d0-801b-91549905dc09-kube-api-access-nz9ld\") pod \"maas-controller-7549975874-7hq4b\" (UID: \"8a80a8ca-48ed-40d0-801b-91549905dc09\") " pod="opendatahub/maas-controller-7549975874-7hq4b"
Sep 11 17:56:14.851252 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:14.851265    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a80a8ca-48ed-40d0-801b-91549905dc09-metrics-certs\") pod \"maas-controller-7549975874-7hq4b\" (UID: \"8a80a8ca-48ed-40d0-801b-91549905dc09\") " pod="opendatahub/maas-controller-7549975874-7hq4b"
Sep 11 17:56:14.851468 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:14.851392    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a80a8ca-48ed-40d0-801b-91549905dc09-webhook-cert\") pod \"maas-controller-7549975874-7hq4b\" (UID: \"8a80a8ca-48ed-40d0-801b-91549905dc09\") " pod="opendatahub/maas-controller-7549975874-7hq4b"
Sep 11 17:56:14.952351 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:14.952322    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a80a8ca-48ed-40d0-801b-91549905dc09-webhook-cert\") pod \"maas-controller-7549975874-7hq4b\" (UID: \"8a80a8ca-48ed-40d0-801b-91549905dc09\") " pod="opendatahub/maas-controller-7549975874-7hq4b"
Sep 11 17:56:14.952351 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:14.952356    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz9ld\" (UniqueName: \"kubernetes.io/projected/8a80a8ca-48ed-40d0-801b-91549905dc09-kube-api-access-nz9ld\") pod \"maas-controller-7549975874-7hq4b\" (UID: \"8a80a8ca-48ed-40d0-801b-91549905dc09\") " pod="opendatahub/maas-controller-7549975874-7hq4b"
Sep 11 17:56:14.952623 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:14.952384    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a80a8ca-48ed-40d0-801b-91549905dc09-metrics-certs\") pod \"maas-controller-7549975874-7hq4b\" (UID: \"8a80a8ca-48ed-40d0-801b-91549905dc09\") " pod="opendatahub/maas-controller-7549975874-7hq4b"
Sep 11 17:56:14.954882 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:14.954859    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a80a8ca-48ed-40d0-801b-91549905dc09-metrics-certs\") pod \"maas-controller-7549975874-7hq4b\" (UID: \"8a80a8ca-48ed-40d0-801b-91549905dc09\") " pod="opendatahub/maas-controller-7549975874-7hq4b"
Sep 11 17:56:14.955015 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:14.954905    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a80a8ca-48ed-40d0-801b-91549905dc09-webhook-cert\") pod \"maas-controller-7549975874-7hq4b\" (UID: \"8a80a8ca-48ed-40d0-801b-91549905dc09\") " pod="opendatahub/maas-controller-7549975874-7hq4b"
Sep 11 17:56:14.961139 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:14.961114    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz9ld\" (UniqueName: \"kubernetes.io/projected/8a80a8ca-48ed-40d0-801b-91549905dc09-kube-api-access-nz9ld\") pod \"maas-controller-7549975874-7hq4b\" (UID: \"8a80a8ca-48ed-40d0-801b-91549905dc09\") " pod="opendatahub/maas-controller-7549975874-7hq4b"
Sep 11 17:56:15.039953 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:15.039863    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7549975874-7hq4b"
Sep 11 17:56:15.312683 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:15.312551    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7549975874-7hq4b"]
Sep 11 17:56:15.314877 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:56:15.314841    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a80a8ca_48ed_40d0_801b_91549905dc09.slice/crio-5491e30684bbaccb3c78fffac328a932f319e2ebcf140592dd09259d06ae8614 WatchSource:0}: Error finding container 5491e30684bbaccb3c78fffac328a932f319e2ebcf140592dd09259d06ae8614: Status 404 returned error can't find the container with id 5491e30684bbaccb3c78fffac328a932f319e2ebcf140592dd09259d06ae8614
Sep 11 17:56:15.924145 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:15.924025    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7549975874-7hq4b" event={"ID":"8a80a8ca-48ed-40d0-801b-91549905dc09","Type":"ContainerStarted","Data":"b9dc512eca47f0a44f94fdc8e415aa34894c59b5dac349f456cda6017c2d49da"}
Sep 11 17:56:15.924145 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:15.924106    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7549975874-7hq4b" event={"ID":"8a80a8ca-48ed-40d0-801b-91549905dc09","Type":"ContainerStarted","Data":"5491e30684bbaccb3c78fffac328a932f319e2ebcf140592dd09259d06ae8614"}
Sep 11 17:56:15.924599 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:15.924229    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7549975874-7hq4b"
Sep 11 17:56:15.954143 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:15.954087    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7549975874-7hq4b" podStartSLOduration=1.6470748560000001 podStartE2EDuration="1.954068225s" totalImagesPullingTime="306.993369ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:56:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:56:15.953309128 +0000 UTC m=+2256.347226583" watchObservedRunningTime="2026-09-11 17:56:15.954068225 +0000 UTC m=+2256.347985681"
Sep 11 17:56:26.931099 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:56:26.931061    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7549975874-7hq4b"
Sep 11 17:59:11.592719 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:59:11.592620    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx"]
Sep 11 17:59:11.596016 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:59:11.595971    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx"
Sep 11 17:59:11.598096 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:59:11.598077    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-31388ae6-kserve-self-signed-certs\""
Sep 11 17:59:11.607859 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:59:11.607835    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx"]
Sep 11 17:59:11.709165 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:59:11.709131    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e-tls-certs\") pod \"e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx\" (UID: \"2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e\") " pod="llm/e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx"
Sep 11 17:59:11.709344 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:59:11.709169    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e-home\") pod \"e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx\" (UID: \"2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e\") " pod="llm/e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx"
Sep 11 17:59:11.709344 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:59:11.709208    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e-tmp-dir\") pod \"e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx\" (UID: \"2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e\") " pod="llm/e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx"
Sep 11 17:59:11.709344 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:59:11.709236    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e-model-cache\") pod \"e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx\" (UID: \"2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e\") " pod="llm/e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx"
Sep 11 17:59:11.709344 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:59:11.709276    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94bkh\" (UniqueName: \"kubernetes.io/projected/2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e-kube-api-access-94bkh\") pod \"e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx\" (UID: \"2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e\") " pod="llm/e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx"
Sep 11 17:59:11.709344 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:59:11.709298    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e-dshm\") pod \"e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx\" (UID: \"2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e\") " pod="llm/e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx"
Sep 11 17:59:11.809944 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:59:11.809906    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94bkh\" (UniqueName: \"kubernetes.io/projected/2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e-kube-api-access-94bkh\") pod \"e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx\" (UID: \"2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e\") " pod="llm/e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx"
Sep 11 17:59:11.809944 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:59:11.809947    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e-dshm\") pod \"e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx\" (UID: \"2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e\") " pod="llm/e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx"
Sep 11 17:59:11.810199 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:59:11.810003    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e-tls-certs\") pod \"e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx\" (UID: \"2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e\") " pod="llm/e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx"
Sep 11 17:59:11.810199 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:59:11.810023    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e-home\") pod \"e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx\" (UID: \"2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e\") " pod="llm/e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx"
Sep 11 17:59:11.810199 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:59:11.810048    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e-tmp-dir\") pod \"e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx\" (UID: \"2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e\") " pod="llm/e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx"
Sep 11 17:59:11.810199 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:59:11.810068    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e-model-cache\") pod \"e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx\" (UID: \"2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e\") " pod="llm/e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx"
Sep 11 17:59:11.810426 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:59:11.810400    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e-tmp-dir\") pod \"e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx\" (UID: \"2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e\") " pod="llm/e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx"
Sep 11 17:59:11.810498 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:59:11.810451    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e-model-cache\") pod \"e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx\" (UID: \"2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e\") " pod="llm/e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx"
Sep 11 17:59:11.810498 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:59:11.810486    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e-home\") pod \"e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx\" (UID: \"2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e\") " pod="llm/e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx"
Sep 11 17:59:11.812399 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:59:11.812377    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e-dshm\") pod \"e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx\" (UID: \"2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e\") " pod="llm/e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx"
Sep 11 17:59:11.812581 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:59:11.812563    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e-tls-certs\") pod \"e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx\" (UID: \"2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e\") " pod="llm/e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx"
Sep 11 17:59:11.818935 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:59:11.818909    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94bkh\" (UniqueName: \"kubernetes.io/projected/2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e-kube-api-access-94bkh\") pod \"e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx\" (UID: \"2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e\") " pod="llm/e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx"
Sep 11 17:59:11.905890 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:59:11.905801    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx"
Sep 11 17:59:12.177192 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:59:12.177111    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx"]
Sep 11 17:59:12.180038 ip-10-0-133-252 kubenswrapper[2562]: W0911 17:59:12.180003    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c4e0af4_6005_40b3_9c2a_a6f6a46c1e4e.slice/crio-8f9b2bbc25e31ed00d32618422ba17d2b0e23085a4434c9749efad71ad342732 WatchSource:0}: Error finding container 8f9b2bbc25e31ed00d32618422ba17d2b0e23085a4434c9749efad71ad342732: Status 404 returned error can't find the container with id 8f9b2bbc25e31ed00d32618422ba17d2b0e23085a4434c9749efad71ad342732
Sep 11 17:59:12.683360 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:59:12.683311    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx" event={"ID":"2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e","Type":"ContainerStarted","Data":"1c4215aa7597053c81b7d05bf263bf920a1252f84574860cd7a7573bb4a1e644"}
Sep 11 17:59:12.683360 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:59:12.683364    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx" event={"ID":"2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e","Type":"ContainerStarted","Data":"8f9b2bbc25e31ed00d32618422ba17d2b0e23085a4434c9749efad71ad342732"}
Sep 11 17:59:12.700575 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:59:12.700529    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx" podStartSLOduration=1.700510679 podStartE2EDuration="1.700510679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:59:12.700346522 +0000 UTC m=+2433.094263978" watchObservedRunningTime="2026-09-11 17:59:12.700510679 +0000 UTC m=+2433.094428132"
Sep 11 17:59:21.906858 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:59:21.906820    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx"
Sep 11 17:59:21.907340 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:59:21.906875    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx"
Sep 11 17:59:21.919672 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:59:21.919642    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx"
Sep 11 17:59:22.744563 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:59:22.744537    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx"
Sep 11 17:59:48.212720 ip-10-0-133-252 kubenswrapper[2562]: I0911 17:59:48.212687    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx"]
Sep 11 18:00:03.231716 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:03.231666    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx" podUID="2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e" containerName="main" containerID="cri-o://1c4215aa7597053c81b7d05bf263bf920a1252f84574860cd7a7573bb4a1e644" gracePeriod=45
Sep 11 18:00:03.472888 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:03.472858    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx"
Sep 11 18:00:03.563469 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:03.563376    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e-dshm\") pod \"2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e\" (UID: \"2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e\") "
Sep 11 18:00:03.563469 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:03.563445    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e-tls-certs\") pod \"2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e\" (UID: \"2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e\") "
Sep 11 18:00:03.563469 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:03.563467    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e-tmp-dir\") pod \"2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e\" (UID: \"2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e\") "
Sep 11 18:00:03.563783 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:03.563495    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94bkh\" (UniqueName: \"kubernetes.io/projected/2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e-kube-api-access-94bkh\") pod \"2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e\" (UID: \"2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e\") "
Sep 11 18:00:03.563783 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:03.563541    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e-home\") pod \"2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e\" (UID: \"2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e\") "
Sep 11 18:00:03.563783 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:03.563572    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e-model-cache\") pod \"2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e\" (UID: \"2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e\") "
Sep 11 18:00:03.563783 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:03.563753    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e" (UID: "2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:00:03.564042 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:03.563864    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e-home" (OuterVolumeSpecName: "home") pod "2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e" (UID: "2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:00:03.564042 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:03.563927    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e-model-cache" (OuterVolumeSpecName: "model-cache") pod "2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e" (UID: "2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:00:03.565764 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:03.565727    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e-dshm" (OuterVolumeSpecName: "dshm") pod "2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e" (UID: "2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:00:03.565869 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:03.565769    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e" (UID: "2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:00:03.565869 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:03.565809    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e-kube-api-access-94bkh" (OuterVolumeSpecName: "kube-api-access-94bkh") pod "2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e" (UID: "2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e"). InnerVolumeSpecName "kube-api-access-94bkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:00:03.664958 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:03.664915    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e-home\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 18:00:03.664958 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:03.664951    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e-model-cache\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 18:00:03.664958 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:03.664961    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e-dshm\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 18:00:03.664958 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:03.664970    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e-tls-certs\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 18:00:03.665272 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:03.664976    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e-tmp-dir\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 18:00:03.665272 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:03.664985    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94bkh\" (UniqueName: \"kubernetes.io/projected/2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e-kube-api-access-94bkh\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 18:00:03.921208 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:03.921173    2562 generic.go:356] "Generic (PLEG): container finished" podID="2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e" containerID="1c4215aa7597053c81b7d05bf263bf920a1252f84574860cd7a7573bb4a1e644" exitCode=0
Sep 11 18:00:03.921389 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:03.921217    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx"
Sep 11 18:00:03.921389 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:03.921258    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx" event={"ID":"2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e","Type":"ContainerDied","Data":"1c4215aa7597053c81b7d05bf263bf920a1252f84574860cd7a7573bb4a1e644"}
Sep 11 18:00:03.921389 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:03.921303    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx" event={"ID":"2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e","Type":"ContainerDied","Data":"8f9b2bbc25e31ed00d32618422ba17d2b0e23085a4434c9749efad71ad342732"}
Sep 11 18:00:03.921389 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:03.921322    2562 scope.go:117] "RemoveContainer" containerID="1c4215aa7597053c81b7d05bf263bf920a1252f84574860cd7a7573bb4a1e644"
Sep 11 18:00:03.930272 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:03.930254    2562 scope.go:117] "RemoveContainer" containerID="1c4215aa7597053c81b7d05bf263bf920a1252f84574860cd7a7573bb4a1e644"
Sep 11 18:00:03.930520 ip-10-0-133-252 kubenswrapper[2562]: E0911 18:00:03.930502    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c4215aa7597053c81b7d05bf263bf920a1252f84574860cd7a7573bb4a1e644\": container with ID starting with 1c4215aa7597053c81b7d05bf263bf920a1252f84574860cd7a7573bb4a1e644 not found: ID does not exist" containerID="1c4215aa7597053c81b7d05bf263bf920a1252f84574860cd7a7573bb4a1e644"
Sep 11 18:00:03.930597 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:03.930528    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c4215aa7597053c81b7d05bf263bf920a1252f84574860cd7a7573bb4a1e644"} err="failed to get container status \"1c4215aa7597053c81b7d05bf263bf920a1252f84574860cd7a7573bb4a1e644\": rpc error: code = NotFound desc = could not find container \"1c4215aa7597053c81b7d05bf263bf920a1252f84574860cd7a7573bb4a1e644\": container with ID starting with 1c4215aa7597053c81b7d05bf263bf920a1252f84574860cd7a7573bb4a1e644 not found: ID does not exist"
Sep 11 18:00:03.942283 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:03.942259    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx"]
Sep 11 18:00:03.946474 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:03.946453    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-31388ae6-kserve-58fcd865f-5hdrx"]
Sep 11 18:00:04.126047 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:04.126015    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e" path="/var/lib/kubelet/pods/2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e/volumes"
Sep 11 18:00:19.264447 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:19.264414    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7dd7c69b97-xxctw"]
Sep 11 18:00:19.265121 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:19.264738    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e" containerName="main"
Sep 11 18:00:19.265121 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:19.264749    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e" containerName="main"
Sep 11 18:00:19.265121 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:19.264829    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c4e0af4-6005-40b3-9c2a-a6f6a46c1e4e" containerName="main"
Sep 11 18:00:19.267719 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:19.267699    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7dd7c69b97-xxctw"
Sep 11 18:00:19.278721 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:19.278696    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7dd7c69b97-xxctw"]
Sep 11 18:00:19.405238 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:19.405207    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05c319f8-c116-4686-8449-3581dcf44537-maas-api-tls\") pod \"maas-api-7dd7c69b97-xxctw\" (UID: \"05c319f8-c116-4686-8449-3581dcf44537\") " pod="odh-ai-gateway-infra/maas-api-7dd7c69b97-xxctw"
Sep 11 18:00:19.405398 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:19.405265    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbbjj\" (UniqueName: \"kubernetes.io/projected/05c319f8-c116-4686-8449-3581dcf44537-kube-api-access-cbbjj\") pod \"maas-api-7dd7c69b97-xxctw\" (UID: \"05c319f8-c116-4686-8449-3581dcf44537\") " pod="odh-ai-gateway-infra/maas-api-7dd7c69b97-xxctw"
Sep 11 18:00:19.405398 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:19.405321    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05c319f8-c116-4686-8449-3581dcf44537-metrics-certs\") pod \"maas-api-7dd7c69b97-xxctw\" (UID: \"05c319f8-c116-4686-8449-3581dcf44537\") " pod="odh-ai-gateway-infra/maas-api-7dd7c69b97-xxctw"
Sep 11 18:00:19.405398 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:19.405366    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05c319f8-c116-4686-8449-3581dcf44537-openshift-service-ca\") pod \"maas-api-7dd7c69b97-xxctw\" (UID: \"05c319f8-c116-4686-8449-3581dcf44537\") " pod="odh-ai-gateway-infra/maas-api-7dd7c69b97-xxctw"
Sep 11 18:00:19.506761 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:19.506721    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05c319f8-c116-4686-8449-3581dcf44537-maas-api-tls\") pod \"maas-api-7dd7c69b97-xxctw\" (UID: \"05c319f8-c116-4686-8449-3581dcf44537\") " pod="odh-ai-gateway-infra/maas-api-7dd7c69b97-xxctw"
Sep 11 18:00:19.506944 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:19.506787    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbbjj\" (UniqueName: \"kubernetes.io/projected/05c319f8-c116-4686-8449-3581dcf44537-kube-api-access-cbbjj\") pod \"maas-api-7dd7c69b97-xxctw\" (UID: \"05c319f8-c116-4686-8449-3581dcf44537\") " pod="odh-ai-gateway-infra/maas-api-7dd7c69b97-xxctw"
Sep 11 18:00:19.506944 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:19.506806    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05c319f8-c116-4686-8449-3581dcf44537-metrics-certs\") pod \"maas-api-7dd7c69b97-xxctw\" (UID: \"05c319f8-c116-4686-8449-3581dcf44537\") " pod="odh-ai-gateway-infra/maas-api-7dd7c69b97-xxctw"
Sep 11 18:00:19.506944 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:19.506832    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05c319f8-c116-4686-8449-3581dcf44537-openshift-service-ca\") pod \"maas-api-7dd7c69b97-xxctw\" (UID: \"05c319f8-c116-4686-8449-3581dcf44537\") " pod="odh-ai-gateway-infra/maas-api-7dd7c69b97-xxctw"
Sep 11 18:00:19.507498 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:19.507464    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05c319f8-c116-4686-8449-3581dcf44537-openshift-service-ca\") pod \"maas-api-7dd7c69b97-xxctw\" (UID: \"05c319f8-c116-4686-8449-3581dcf44537\") " pod="odh-ai-gateway-infra/maas-api-7dd7c69b97-xxctw"
Sep 11 18:00:19.509242 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:19.509220    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05c319f8-c116-4686-8449-3581dcf44537-metrics-certs\") pod \"maas-api-7dd7c69b97-xxctw\" (UID: \"05c319f8-c116-4686-8449-3581dcf44537\") " pod="odh-ai-gateway-infra/maas-api-7dd7c69b97-xxctw"
Sep 11 18:00:19.509342 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:19.509276    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05c319f8-c116-4686-8449-3581dcf44537-maas-api-tls\") pod \"maas-api-7dd7c69b97-xxctw\" (UID: \"05c319f8-c116-4686-8449-3581dcf44537\") " pod="odh-ai-gateway-infra/maas-api-7dd7c69b97-xxctw"
Sep 11 18:00:19.518656 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:19.518597    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbbjj\" (UniqueName: \"kubernetes.io/projected/05c319f8-c116-4686-8449-3581dcf44537-kube-api-access-cbbjj\") pod \"maas-api-7dd7c69b97-xxctw\" (UID: \"05c319f8-c116-4686-8449-3581dcf44537\") " pod="odh-ai-gateway-infra/maas-api-7dd7c69b97-xxctw"
Sep 11 18:00:19.579141 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:19.579107    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7dd7c69b97-xxctw"
Sep 11 18:00:19.853026 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:19.852983    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7dd7c69b97-xxctw"]
Sep 11 18:00:19.855107 ip-10-0-133-252 kubenswrapper[2562]: W0911 18:00:19.855080    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05c319f8_c116_4686_8449_3581dcf44537.slice/crio-7e8de157c753c8e423891a74152e0338ec3ba9ec8a4e46997b7a0047db8411b7 WatchSource:0}: Error finding container 7e8de157c753c8e423891a74152e0338ec3ba9ec8a4e46997b7a0047db8411b7: Status 404 returned error can't find the container with id 7e8de157c753c8e423891a74152e0338ec3ba9ec8a4e46997b7a0047db8411b7
Sep 11 18:00:19.994973 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:19.994934    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7dd7c69b97-xxctw" event={"ID":"05c319f8-c116-4686-8449-3581dcf44537","Type":"ContainerStarted","Data":"7e8de157c753c8e423891a74152e0338ec3ba9ec8a4e46997b7a0047db8411b7"}
Sep 11 18:00:21.001449 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:21.001404    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7dd7c69b97-xxctw" event={"ID":"05c319f8-c116-4686-8449-3581dcf44537","Type":"ContainerStarted","Data":"ee4045814bfa07b7828dd6e1830a2126d0b1a0fb4ccec16571c10f2bdc70c89e"}
Sep 11 18:00:21.001904 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:21.001480    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7dd7c69b97-xxctw"
Sep 11 18:00:21.023179 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:21.023101    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7dd7c69b97-xxctw" podStartSLOduration=1.6202673829999998 podStartE2EDuration="2.023079077s" totalImagesPullingTime="402.811694ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:00:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:00:21.022513072 +0000 UTC m=+2501.416430526" watchObservedRunningTime="2026-09-11 18:00:21.023079077 +0000 UTC m=+2501.416996531"
Sep 11 18:00:27.011516 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:27.011486    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7dd7c69b97-xxctw"
Sep 11 18:00:27.065884 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:27.065841    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ddbcf49fc-j64kl"]
Sep 11 18:00:27.066104 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:27.066081    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5ddbcf49fc-j64kl" podUID="837bcd38-4699-4d31-b335-ce9da7007b57" containerName="maas-api" containerID="cri-o://94b15785308aedd3985f07c841b2f65db7a25be612a412a8ff0beb6c1d12e1a9" gracePeriod=30
Sep 11 18:00:28.311763 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:28.311736    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ddbcf49fc-j64kl"
Sep 11 18:00:28.386085 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:28.386053    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4tlt\" (UniqueName: \"kubernetes.io/projected/837bcd38-4699-4d31-b335-ce9da7007b57-kube-api-access-g4tlt\") pod \"837bcd38-4699-4d31-b335-ce9da7007b57\" (UID: \"837bcd38-4699-4d31-b335-ce9da7007b57\") "
Sep 11 18:00:28.386288 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:28.386103    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/837bcd38-4699-4d31-b335-ce9da7007b57-metrics-certs\") pod \"837bcd38-4699-4d31-b335-ce9da7007b57\" (UID: \"837bcd38-4699-4d31-b335-ce9da7007b57\") "
Sep 11 18:00:28.386288 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:28.386152    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/837bcd38-4699-4d31-b335-ce9da7007b57-openshift-service-ca\") pod \"837bcd38-4699-4d31-b335-ce9da7007b57\" (UID: \"837bcd38-4699-4d31-b335-ce9da7007b57\") "
Sep 11 18:00:28.386288 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:28.386183    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/837bcd38-4699-4d31-b335-ce9da7007b57-maas-api-tls\") pod \"837bcd38-4699-4d31-b335-ce9da7007b57\" (UID: \"837bcd38-4699-4d31-b335-ce9da7007b57\") "
Sep 11 18:00:28.386600 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:28.386559    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/837bcd38-4699-4d31-b335-ce9da7007b57-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "837bcd38-4699-4d31-b335-ce9da7007b57" (UID: "837bcd38-4699-4d31-b335-ce9da7007b57"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:00:28.388276 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:28.388251    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/837bcd38-4699-4d31-b335-ce9da7007b57-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "837bcd38-4699-4d31-b335-ce9da7007b57" (UID: "837bcd38-4699-4d31-b335-ce9da7007b57"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:00:28.388372 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:28.388344    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/837bcd38-4699-4d31-b335-ce9da7007b57-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "837bcd38-4699-4d31-b335-ce9da7007b57" (UID: "837bcd38-4699-4d31-b335-ce9da7007b57"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:00:28.388413 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:28.388388    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/837bcd38-4699-4d31-b335-ce9da7007b57-kube-api-access-g4tlt" (OuterVolumeSpecName: "kube-api-access-g4tlt") pod "837bcd38-4699-4d31-b335-ce9da7007b57" (UID: "837bcd38-4699-4d31-b335-ce9da7007b57"). InnerVolumeSpecName "kube-api-access-g4tlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:00:28.487845 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:28.487768    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/837bcd38-4699-4d31-b335-ce9da7007b57-maas-api-tls\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 18:00:28.487845 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:28.487797    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4tlt\" (UniqueName: \"kubernetes.io/projected/837bcd38-4699-4d31-b335-ce9da7007b57-kube-api-access-g4tlt\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 18:00:28.487845 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:28.487821    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/837bcd38-4699-4d31-b335-ce9da7007b57-metrics-certs\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 18:00:28.487845 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:28.487830    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/837bcd38-4699-4d31-b335-ce9da7007b57-openshift-service-ca\") on node \"ip-10-0-133-252.ec2.internal\" DevicePath \"\""
Sep 11 18:00:29.044336 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:29.044296    2562 generic.go:356] "Generic (PLEG): container finished" podID="837bcd38-4699-4d31-b335-ce9da7007b57" containerID="94b15785308aedd3985f07c841b2f65db7a25be612a412a8ff0beb6c1d12e1a9" exitCode=0
Sep 11 18:00:29.044527 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:29.044351    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ddbcf49fc-j64kl"
Sep 11 18:00:29.044527 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:29.044349    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ddbcf49fc-j64kl" event={"ID":"837bcd38-4699-4d31-b335-ce9da7007b57","Type":"ContainerDied","Data":"94b15785308aedd3985f07c841b2f65db7a25be612a412a8ff0beb6c1d12e1a9"}
Sep 11 18:00:29.044527 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:29.044471    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ddbcf49fc-j64kl" event={"ID":"837bcd38-4699-4d31-b335-ce9da7007b57","Type":"ContainerDied","Data":"5e4b523fc5820760734ae410b25bdb3ff985216177727a1d5d2fcfe7f2fdafea"}
Sep 11 18:00:29.044527 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:29.044494    2562 scope.go:117] "RemoveContainer" containerID="94b15785308aedd3985f07c841b2f65db7a25be612a412a8ff0beb6c1d12e1a9"
Sep 11 18:00:29.053953 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:29.053929    2562 scope.go:117] "RemoveContainer" containerID="94b15785308aedd3985f07c841b2f65db7a25be612a412a8ff0beb6c1d12e1a9"
Sep 11 18:00:29.054232 ip-10-0-133-252 kubenswrapper[2562]: E0911 18:00:29.054213    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94b15785308aedd3985f07c841b2f65db7a25be612a412a8ff0beb6c1d12e1a9\": container with ID starting with 94b15785308aedd3985f07c841b2f65db7a25be612a412a8ff0beb6c1d12e1a9 not found: ID does not exist" containerID="94b15785308aedd3985f07c841b2f65db7a25be612a412a8ff0beb6c1d12e1a9"
Sep 11 18:00:29.054290 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:29.054241    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94b15785308aedd3985f07c841b2f65db7a25be612a412a8ff0beb6c1d12e1a9"} err="failed to get container status \"94b15785308aedd3985f07c841b2f65db7a25be612a412a8ff0beb6c1d12e1a9\": rpc error: code = NotFound desc = could not find container \"94b15785308aedd3985f07c841b2f65db7a25be612a412a8ff0beb6c1d12e1a9\": container with ID starting with 94b15785308aedd3985f07c841b2f65db7a25be612a412a8ff0beb6c1d12e1a9 not found: ID does not exist"
Sep 11 18:00:29.068304 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:29.068278    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ddbcf49fc-j64kl"]
Sep 11 18:00:29.074150 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:29.074125    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ddbcf49fc-j64kl"]
Sep 11 18:00:30.129472 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:30.129440    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="837bcd38-4699-4d31-b335-ce9da7007b57" path="/var/lib/kubelet/pods/837bcd38-4699-4d31-b335-ce9da7007b57/volumes"
Sep 11 18:00:48.208672 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:48.208642    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-8bbvh_eca3874b-6600-44d9-99a3-4ee52e6e1c6b/manager/0.log"
Sep 11 18:00:48.503920 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:48.503840    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7549975874-7hq4b_8a80a8ca-48ed-40d0-801b-91549905dc09/manager/0.log"
Sep 11 18:00:49.173152 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:49.173124    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b887fdd9f-pnsph_4a57bf87-8a95-4d08-b354-14cc791deb5a/copy-manifests/0.log"
Sep 11 18:00:49.190933 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:49.190903    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b887fdd9f-pnsph_4a57bf87-8a95-4d08-b354-14cc791deb5a/manager/0.log"
Sep 11 18:00:49.701644 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:49.701608    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7dd7c69b97-xxctw_05c319f8-c116-4686-8449-3581dcf44537/maas-api/0.log"
Sep 11 18:00:50.191320 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:50.191290    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-bqkl9_389f217e-71c7-40d9-99f7-730d7b7534b4/postgres/0.log"
Sep 11 18:00:51.224619 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:51.224582    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-sn5c6_9bdbe8b0-12cc-4c64-873c-bf6fc3106b40/manager/0.log"
Sep 11 18:00:51.337877 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:51.337848    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-r7gz4_c2da120d-b822-45c9-9881-a9dac929a8d2/manager/0.log"
Sep 11 18:00:52.430512 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:52.430479    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-rp5zl_9f26863e-47a2-44f5-9ba9-6e7643bbf847/discovery/0.log"
Sep 11 18:00:52.686370 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:52.686286    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d6df6ff46-slct7_39f092d7-877a-4372-8f38-a32be1e7bd46/kube-auth-proxy/0.log"
Sep 11 18:00:52.803738 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:52.803707    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4ktwc_59094ae7-d124-40e9-be6a-59a658cd635d/istio-proxy/1.log"
Sep 11 18:00:53.039889 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:53.039813    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-lr9nq_5f6b61e9-0864-47bb-aeec-df01bbf50d34/payload-processing/0.log"
Sep 11 18:00:53.648440 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:53.648402    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-gndz7_cecd860f-bb94-43b0-93f8-967f4a227add/main/0.log"
Sep 11 18:00:54.004747 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:00:54.004668    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f67b9f_31de6239-94e0-410b-97fd-a613b33b12c2/main/0.log"
Sep 11 18:01:03.654617 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:03.654582    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4lmxn_85715083-7220-4a0d-9dd0-426a131d8440/global-pull-secret-syncer/0.log"
Sep 11 18:01:03.874820 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:03.874788    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qjdvg_edbdd0ae-4406-4088-9230-c68aed929e7b/konnectivity-agent/0.log"
Sep 11 18:01:03.899365 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:03.899337    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-252.ec2.internal_9202eb18634ea6b37b4f9ec563ddc4eb/haproxy/0.log"
Sep 11 18:01:09.136680 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:09.136649    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-sn5c6_9bdbe8b0-12cc-4c64-873c-bf6fc3106b40/manager/0.log"
Sep 11 18:01:09.163508 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:09.163478    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-r7gz4_c2da120d-b822-45c9-9881-a9dac929a8d2/manager/0.log"
Sep 11 18:01:11.183107 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:11.183065    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6xfx5_d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d/node-exporter/0.log"
Sep 11 18:01:11.209118 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:11.209086    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6xfx5_d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d/kube-rbac-proxy/0.log"
Sep 11 18:01:11.234134 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:11.234054    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6xfx5_d3a189bc-8f76-4ad7-95aa-5d006ca0dc3d/init-textfile/0.log"
Sep 11 18:01:13.977590 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:13.977536    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tsftd/perf-node-gather-daemonset-t5cdf"]
Sep 11 18:01:13.977984 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:13.977889    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="837bcd38-4699-4d31-b335-ce9da7007b57" containerName="maas-api"
Sep 11 18:01:13.977984 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:13.977901    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="837bcd38-4699-4d31-b335-ce9da7007b57" containerName="maas-api"
Sep 11 18:01:13.977984 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:13.977954    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="837bcd38-4699-4d31-b335-ce9da7007b57" containerName="maas-api"
Sep 11 18:01:13.981101 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:13.981048    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-t5cdf"
Sep 11 18:01:13.983420 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:13.983400    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tsftd\"/\"openshift-service-ca.crt\""
Sep 11 18:01:13.983687 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:13.983672    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tsftd\"/\"default-dockercfg-cw6vd\""
Sep 11 18:01:13.984233 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:13.984219    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tsftd\"/\"kube-root-ca.crt\""
Sep 11 18:01:13.991729 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:13.991703    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tsftd/perf-node-gather-daemonset-t5cdf"]
Sep 11 18:01:14.103425 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:14.103394    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3f0d1e35-33f7-4f56-a8a9-5e562700b861-lib-modules\") pod \"perf-node-gather-daemonset-t5cdf\" (UID: \"3f0d1e35-33f7-4f56-a8a9-5e562700b861\") " pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-t5cdf"
Sep 11 18:01:14.103631 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:14.103442    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3f0d1e35-33f7-4f56-a8a9-5e562700b861-podres\") pod \"perf-node-gather-daemonset-t5cdf\" (UID: \"3f0d1e35-33f7-4f56-a8a9-5e562700b861\") " pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-t5cdf"
Sep 11 18:01:14.103631 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:14.103498    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3f0d1e35-33f7-4f56-a8a9-5e562700b861-proc\") pod \"perf-node-gather-daemonset-t5cdf\" (UID: \"3f0d1e35-33f7-4f56-a8a9-5e562700b861\") " pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-t5cdf"
Sep 11 18:01:14.103631 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:14.103561    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f0d1e35-33f7-4f56-a8a9-5e562700b861-sys\") pod \"perf-node-gather-daemonset-t5cdf\" (UID: \"3f0d1e35-33f7-4f56-a8a9-5e562700b861\") " pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-t5cdf"
Sep 11 18:01:14.103631 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:14.103588    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr5hd\" (UniqueName: \"kubernetes.io/projected/3f0d1e35-33f7-4f56-a8a9-5e562700b861-kube-api-access-nr5hd\") pod \"perf-node-gather-daemonset-t5cdf\" (UID: \"3f0d1e35-33f7-4f56-a8a9-5e562700b861\") " pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-t5cdf"
Sep 11 18:01:14.204318 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:14.204283    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3f0d1e35-33f7-4f56-a8a9-5e562700b861-lib-modules\") pod \"perf-node-gather-daemonset-t5cdf\" (UID: \"3f0d1e35-33f7-4f56-a8a9-5e562700b861\") " pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-t5cdf"
Sep 11 18:01:14.204589 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:14.204352    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3f0d1e35-33f7-4f56-a8a9-5e562700b861-podres\") pod \"perf-node-gather-daemonset-t5cdf\" (UID: \"3f0d1e35-33f7-4f56-a8a9-5e562700b861\") " pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-t5cdf"
Sep 11 18:01:14.204589 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:14.204376    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3f0d1e35-33f7-4f56-a8a9-5e562700b861-proc\") pod \"perf-node-gather-daemonset-t5cdf\" (UID: \"3f0d1e35-33f7-4f56-a8a9-5e562700b861\") " pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-t5cdf"
Sep 11 18:01:14.204589 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:14.204469    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f0d1e35-33f7-4f56-a8a9-5e562700b861-sys\") pod \"perf-node-gather-daemonset-t5cdf\" (UID: \"3f0d1e35-33f7-4f56-a8a9-5e562700b861\") " pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-t5cdf"
Sep 11 18:01:14.204589 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:14.204476    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3f0d1e35-33f7-4f56-a8a9-5e562700b861-lib-modules\") pod \"perf-node-gather-daemonset-t5cdf\" (UID: \"3f0d1e35-33f7-4f56-a8a9-5e562700b861\") " pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-t5cdf"
Sep 11 18:01:14.204589 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:14.204497    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3f0d1e35-33f7-4f56-a8a9-5e562700b861-podres\") pod \"perf-node-gather-daemonset-t5cdf\" (UID: \"3f0d1e35-33f7-4f56-a8a9-5e562700b861\") " pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-t5cdf"
Sep 11 18:01:14.204589 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:14.204484    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3f0d1e35-33f7-4f56-a8a9-5e562700b861-proc\") pod \"perf-node-gather-daemonset-t5cdf\" (UID: \"3f0d1e35-33f7-4f56-a8a9-5e562700b861\") " pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-t5cdf"
Sep 11 18:01:14.204589 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:14.204516    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr5hd\" (UniqueName: \"kubernetes.io/projected/3f0d1e35-33f7-4f56-a8a9-5e562700b861-kube-api-access-nr5hd\") pod \"perf-node-gather-daemonset-t5cdf\" (UID: \"3f0d1e35-33f7-4f56-a8a9-5e562700b861\") " pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-t5cdf"
Sep 11 18:01:14.204589 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:14.204580    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f0d1e35-33f7-4f56-a8a9-5e562700b861-sys\") pod \"perf-node-gather-daemonset-t5cdf\" (UID: \"3f0d1e35-33f7-4f56-a8a9-5e562700b861\") " pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-t5cdf"
Sep 11 18:01:14.212552 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:14.212532    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr5hd\" (UniqueName: \"kubernetes.io/projected/3f0d1e35-33f7-4f56-a8a9-5e562700b861-kube-api-access-nr5hd\") pod \"perf-node-gather-daemonset-t5cdf\" (UID: \"3f0d1e35-33f7-4f56-a8a9-5e562700b861\") " pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-t5cdf"
Sep 11 18:01:14.292452 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:14.292362    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-t5cdf"
Sep 11 18:01:14.567058 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:14.567028    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tsftd/perf-node-gather-daemonset-t5cdf"]
Sep 11 18:01:14.569205 ip-10-0-133-252 kubenswrapper[2562]: W0911 18:01:14.569169    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3f0d1e35_33f7_4f56_a8a9_5e562700b861.slice/crio-f530b1f8db0708c37792f416e10b0169267a163453443a9ca12538a08adfcf02 WatchSource:0}: Error finding container f530b1f8db0708c37792f416e10b0169267a163453443a9ca12538a08adfcf02: Status 404 returned error can't find the container with id f530b1f8db0708c37792f416e10b0169267a163453443a9ca12538a08adfcf02
Sep 11 18:01:15.244588 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:15.244553    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-t5cdf" event={"ID":"3f0d1e35-33f7-4f56-a8a9-5e562700b861","Type":"ContainerStarted","Data":"25ae34f3e8c44f7f1ff18c5e87e30f4ba106c9379fe7113d37054302b62c9a37"}
Sep 11 18:01:15.245046 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:15.244606    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-t5cdf" event={"ID":"3f0d1e35-33f7-4f56-a8a9-5e562700b861","Type":"ContainerStarted","Data":"f530b1f8db0708c37792f416e10b0169267a163453443a9ca12538a08adfcf02"}
Sep 11 18:01:15.245046 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:15.244665    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-t5cdf"
Sep 11 18:01:15.271919 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:15.271846    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-t5cdf" podStartSLOduration=2.271825324 podStartE2EDuration="2.271825324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:01:15.271607738 +0000 UTC m=+2555.665525194" watchObservedRunningTime="2026-09-11 18:01:15.271825324 +0000 UTC m=+2555.665742780"
Sep 11 18:01:16.651111 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:16.651079    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z4bx7_77f356e7-bfb0-47dc-9910-4d49ef7f2d71/dns/0.log"
Sep 11 18:01:16.672960 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:16.672923    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z4bx7_77f356e7-bfb0-47dc-9910-4d49ef7f2d71/kube-rbac-proxy/0.log"
Sep 11 18:01:16.749619 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:16.749587    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-km5mq_c3c72203-332a-4273-8607-6e7f71377e54/dns-node-resolver/0.log"
Sep 11 18:01:17.278322 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:17.278290    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-647bdcc9cf-tsrbr_76c521a8-16b1-487f-ae42-3f94d0605fcb/registry/0.log"
Sep 11 18:01:17.303372 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:17.303315    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2sn6n_924a846a-540b-48c3-a5d7-8df3fbaa41e8/node-ca/0.log"
Sep 11 18:01:19.120511 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:19.120472    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-rp5zl_9f26863e-47a2-44f5-9ba9-6e7643bbf847/discovery/0.log"
Sep 11 18:01:19.179915 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:19.179883    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d6df6ff46-slct7_39f092d7-877a-4372-8f38-a32be1e7bd46/kube-auth-proxy/0.log"
Sep 11 18:01:19.215395 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:19.215355    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4ktwc_59094ae7-d124-40e9-be6a-59a658cd635d/istio-proxy/0.log"
Sep 11 18:01:19.220248 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:19.220223    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4ktwc_59094ae7-d124-40e9-be6a-59a658cd635d/istio-proxy/1.log"
Sep 11 18:01:19.292153 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:19.292117    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-lr9nq_5f6b61e9-0864-47bb-aeec-df01bbf50d34/payload-processing/0.log"
Sep 11 18:01:20.262739 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:20.262709    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jd95w_f87ec688-4995-4c18-9ee4-fe5afda82b21/serve-healthcheck-canary/0.log"
Sep 11 18:01:20.887245 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:20.887216    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gnrll_daabf0e8-8fd4-4a7d-85b5-1713702add55/kube-rbac-proxy/0.log"
Sep 11 18:01:20.909835 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:20.909808    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gnrll_daabf0e8-8fd4-4a7d-85b5-1713702add55/exporter/0.log"
Sep 11 18:01:20.933652 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:20.933621    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gnrll_daabf0e8-8fd4-4a7d-85b5-1713702add55/extractor/0.log"
Sep 11 18:01:21.259344 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:21.259271    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tsftd/perf-node-gather-daemonset-t5cdf"
Sep 11 18:01:23.479741 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:23.479655    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-8bbvh_eca3874b-6600-44d9-99a3-4ee52e6e1c6b/manager/0.log"
Sep 11 18:01:24.101034 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:24.100977    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7549975874-7hq4b_8a80a8ca-48ed-40d0-801b-91549905dc09/manager/0.log"
Sep 11 18:01:24.526870 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:24.526835    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b887fdd9f-pnsph_4a57bf87-8a95-4d08-b354-14cc791deb5a/manager/0.log"
Sep 11 18:01:24.546683 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:24.546661    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b887fdd9f-pnsph_4a57bf87-8a95-4d08-b354-14cc791deb5a/copy-manifests/0.log"
Sep 11 18:01:25.569901 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:25.569818    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-rp5zl_9f26863e-47a2-44f5-9ba9-6e7643bbf847/discovery/0.log"
Sep 11 18:01:25.596582 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:25.596554    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d6df6ff46-slct7_39f092d7-877a-4372-8f38-a32be1e7bd46/kube-auth-proxy/0.log"
Sep 11 18:01:25.618785 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:25.618761    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4ktwc_59094ae7-d124-40e9-be6a-59a658cd635d/istio-proxy/0.log"
Sep 11 18:01:25.623253 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:25.623232    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4ktwc_59094ae7-d124-40e9-be6a-59a658cd635d/istio-proxy/1.log"
Sep 11 18:01:25.676287 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:25.676260    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-lr9nq_5f6b61e9-0864-47bb-aeec-df01bbf50d34/payload-processing/0.log"
Sep 11 18:01:28.124575 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:28.124547    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-rp5zl_9f26863e-47a2-44f5-9ba9-6e7643bbf847/discovery/0.log"
Sep 11 18:01:28.165641 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:28.165608    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d6df6ff46-slct7_39f092d7-877a-4372-8f38-a32be1e7bd46/kube-auth-proxy/0.log"
Sep 11 18:01:28.187677 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:28.187650    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4ktwc_59094ae7-d124-40e9-be6a-59a658cd635d/istio-proxy/0.log"
Sep 11 18:01:28.193736 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:28.193711    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4ktwc_59094ae7-d124-40e9-be6a-59a658cd635d/istio-proxy/1.log"
Sep 11 18:01:28.247216 ip-10-0-133-252 kubenswrapper[2562]: I0911 18:01:28.247177    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-lr9nq_5f6b61e9-0864-47bb-aeec-df01bbf50d34/payload-processing/0.log"